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PREFACE 


The  Navy  Junior  Reserve  Officers  Training  Corps  (NJROTC)  provides 
academic  instruction  and  leadership  training  to  high  school  stu¬ 
dents.  Approximately  ten  years  ago,  a  computer-based  multimedia 
instructional  system  was  developed  in  the  NJROTC  unit  at  Sanger 
High  School  in  California.  Over  the  years,  the  system  has  been  re¬ 
fined  and  it  has  been  installed  at  other  schools.  Before  determining 
whether  to  expand  the  use  of  the  system  to  additional  NJROTC  loca¬ 
tions,  the  Office  of  the  Chief  of  Naval  Education  and  Training  (CNET) 
asked  RAND’s  National  Defense  Research  Institute  (NDRI)  to  evalu¬ 
ate  it. 

This  study  was  conducted  at  two  high  schools.  It  is  an  “exploratory” 
study  in  that  pre-existing  groups  were  used  instead  of  randomly  as¬ 
signed  groups,  and  tests  were  not  conducted  to  determine  whether 
the  students,  instructors  and  environmental  factors  at  those  schools 
are  typical  of  NJROTC  units.  Consequently,  the  findings  are  infor¬ 
mative,  but  they  should  not  be  generalized  to  other  locations. 

We  found  that  students  who  used  the  system  made  greater  academic 
progress  than  their  peers  who  received  traditional  methods  of  in¬ 
struction.  During  focus  groups,  students  reported  that  classes  using 
the  system  were  better  organized  and  more  compelling  than  tradi¬ 
tional  classes.  Instructors  reported  that  the  system  reduced  their 
workload  and  facilitated  their  ability  to  enhance  the  course. 

This  study  suggests  explanations  for  the  apparent  efficacy  of  the  sys¬ 
tem.  It  also  discusses  the  potential  for  extending  the  system  beyond 
its  current  applications. 
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The  work  reported  here  was  sponsored  by  CNET  and  was  carried  out 
within  the  Forces  and  Resources  Policy  Center  of  RAND’s  NDRI. 
NDRI  is  a  federally  funded  research  and  development  center  spon¬ 
sored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  and 
the  defense  agencies. 
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SUMMARY 


The  Navy  Junior  Reserve  Officers  Training  Corps  (NJROTC)  provides 
high  school  students  with  academic  training  and  military  leadership 
experience.  The  academic  portion  of  the  program  consists  of  a  four- 
year  Naval  Science  curriculum  that  includes  naval  history,  sea  power, 
maritime  geography,  navigation,  citizenship,  oceanography,  meteo¬ 
rology,  military  justice,  astronomy,  naval  operations,  communica¬ 
tions,  and  intelligence.  The  Office  of  the  Chief  of  Naval  Education 
and  Training  (CNET)  provides  most  of  the  course  materials  (i.e.,  in¬ 
structors’  guides,  textbooks,  audiovisual  aids,  and  tests). 

Approximately  ten  years  ago,  a  computer-based  NJROTC  multimedia 
instructional  system  (NMIS)  was  developed  in  the  NJROTC  program 
at  Sanger  High  School,  Sanger,  California,  to  enhance  the  delivery  of 
the  Naval  Science  curriculum.  The  system  has  undergone  extensive 
and  relatively  continuous  refinement  since  its  inception,  and  it  has 
been  adopted  by  other  schools. 

THE  NJROTC  MULTIMEDIA  INSTRUCTIONAL  SYSTEM 

Physically,  the  principal  components  of  the  NMIS  are  an  instructor’s 
computer,  a  large  television  monitor  located  at  the  front  of  the  class¬ 
room,  and  individual  computers  for  28  students;  all  are  connected 
via  a  local  area  network.  Functionally,  the  system  can  be  defined 
principally  in  terms  of  course  lectures,  reading  materials,  and  tests. 

During  lectures,  the  system  is  used  to  present  key  points  of  the  lesson 
via  “Instructor’s  Notes,”  which  are  projected  on  the  screen  at  the 
front  of  the  room  (in  the  manner  that  slides  or  transparencies  are 
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used  in  military  briefings  and  during  meetings  in  business  and  other 
organizations).  The  copy  appearing  on  the  instructor’s  computer 
screen  includes  ’buttons”  which,  when  clicked,  activate  auxiliary 
equipment  (such  as  the  laser  disk  player  provided  by  CNET);  in  this 
manner,  still  pictures,  moving  pictures,  graphics,  audio  clips,  etc.,  are 
incorporated  easily  into  the  lecture.  An  abbreviated  version  of  the 
Instructor’s  Notes  appears  on  the  students’  screens,  and  they  take 
notes  by  annotating  that  copy. 

Students  read  course  materials,  such  as  the  textbook,  on  their  com¬ 
puter  screen.  The  electronic  version  of  the  textbook  has  hypertext 
links  to  pictures,  other  graphics,  and  a  glossary.  A  split  screen  is  used 
during  the  reading  sessions;  reading  material  appears  in  a  window 
on  the  left  side  of  the  screen;  students  take  notes  in  a  window  on  the 
right  side. 

The  system  also  is  used  to  administer  tests.  Tests  can  be  given  in  a 
variety  of  formats  including  multiple  choice,  true/false,  brief  essay, 
and  graphical  (e.g.,  “Identify  ‘Country  A’  on  the  map  displayed 
above.”).  The  system  informs  students  of  the  correct  answer  to  a 
question  when  they  finalize  their  response,  and  it  gives  them  their 
score  when  they  complete  the  exam.  At  the  end  of  the  exam  period, 
the  system  prepares  statistical  reports  for  the  instructor  (e.g.,  results 
by  student  or  by  question,  as  raw  scores,  as  percentages,  and  so 
forth). 

It  has  been  posited  that  the  NMIS  promotes  student  achievement 
and  enhances  their  attitudes  toward  courses  through  mechanisms 
such  as  multiple  representations  of  material,  immediate  feedback, 
and  aids  that  facilitate  note-taking.  Similarly,  system  features  that 
organize  the  lectures,  integrate  audio  and  visual  aids,  and  score  stu¬ 
dents’  tests  were  posited  to  have  a  positive  effect  on  the  instructors 
workload. 

THE  RESEARCH  QUESTIONS 

Before  determining  whether  to  install  copies  of  the  NMIS  at  addi¬ 
tional  NJROTC  units,  CNET  turned  to  RAND’s  National  Defense  Re¬ 
search  Institute  (NDRI)  to  evaluate  the  system.  We  were  asked  to 
explore  a  number  of  topics,  which  we  synthesized  into  three  broad 
research  questions: 
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•  Does  the  system  have  a  positive  effect  on  students’  aca¬ 
demic  achievement? 

•  What  are  the  students'  attitudes  toward  courses  using 
the  system? 

•  What  effect,  if  any,  does  the  system  have  on  instructors’ 
workload? 

METHODOLOGY 

We  conducted  this  exploratory  study  at  Sanger  and  Lemoore  High 
Schools  over  a  ten-week  period  during  the  fall  term  of  the  1995-1996 
school  year.  Because  of  logistical  and  time  constraints,  it  was  not 
feasible  to  assign  students  to  experimental  and  control  groups. 
However,  naturally  occurring  comparison  groups  were  identified  in 
three  courses.  Each  of  those  courses  consisted  of  multiple  sections 
to  which  students  had  been  assigned  before  the  start  of  the  study. 
Some  sections  used  the  system  and  some  sections  were  taught  using 
traditional  methods  of  instruction. 

Students’  scores  on  a  pre-test  and  post-test  were  used  to  measure  the 
effect  of  the  NMIS  on  students'  achievement.  Comments  collected 
during  11  focus  group  sessions  with  124  students  at  Lemoore  and 
Sanger  were  used  to  examine  students’  attitudes  toward  courses  in¬ 
corporating  the  system.  Interviews  with  the  instructors  explored  the 
effect  of  the  system  on  instructors’  workloads.  In  addition,  we  ob¬ 
served  classes  to  gather  information  that  would  supplement  and  il¬ 
luminate  the  principal  sources  of  data.  We  also  collected  other  data, 
such  as  students’  scores  on  standardized  tests,  to  determine  the 
comparability  of  the  students  in  both  groups  of  each  course  (i.e.,  stu¬ 
dents  using  the  NMIS,  and  students  being  instructed  with  traditional 
methods). 

FINDINGS 

The  data  indicate  that  the  NMIS  has  a  positive  effect  on  students’ 
academic  achievement.  In  each  of  the  three  courses  that  partici¬ 
pated  in  this  study,  students  in  the  NMIS  group  achieved  greater  per¬ 
centage  gains  from  the  pre-test  to  the  post-test  than  did  students  in 
the  group  using  traditional  methods  of  instruction. 
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Students  reported  that  multiple  representations  of  the  course  mate 
rial  help  them  understand  it  better  than,  for  example,  just  hearing  an 
instructor  talk  about  it;  immediate  feedback  during  the  tests  is  help¬ 
ful;  and  mechanical  aids  imbedded  in  the  system  improve  students’ 
productivity.  In  addition,  they  commented  that,  relative  to  courses 
using  traditional  methods  of  instruction,  (1)  lectures  are  better  or¬ 
ganized,  consequently  they  are  easier  to  follow  and  there  is  more 
time  during  the  class  period  for  discussion  and  students’  questions, 
(2)  course  reading  materials  are  more  compelling  than  ordinary  text¬ 
books  because  the  system  interacts  with  the  students,  e.g.,  it  locates 
material  instantly  and  provides  hypertext  links  to  pictures  and  a  glos¬ 
sary,  and  (3)  instructors  are  more  likely  to  supplement  standard 
course  material  with,  for  example,  articles  from  the  morning’s  news¬ 
paper,  which  makes  the  lessons  more  relevant  and  more  interesting. 

Data  collected  during  interviews  with  the  instructors  indicate  that 
the  NMIS  has  a  favorable  effect  on  their  workload.  More  specifically, 
relative  to  traditional  methods  of  instruction,  (1)  lessons  are  supple¬ 
mented,  updated,  and  corrected  more  easily  with  the  NMIS,  (2)  an 
“enormous”  number  of  hours  of  clerical  work  have  been  eliminated 
because  most  of  the  tests  are  scored  entirely  by  the  system,  (3)  less 
preparation  is  required  before  class  because  lecture  materials  are 
better  organized,  (4)  less  class  time  is  required  to  deal  with  routine 
administrative  matters  such  as  collecting  and  returning  students 
papers,  etc.,  (5)  less  class  time  is  needed  to  prepare  students  to  re¬ 
ceive  the  lesson  (i.e.,  to  “settle  down,”  get  out  their  paper,  pen,  and 
textbook,  and  turn  to  the  designated  chapter),  and  (6)  less  class  time 
is  taken  to  set  up  and  operate  audio  and  visual  aid  equipment  be¬ 
cause,  with  the  system,  the  equipment  is  activated  by  clicking 
“buttons”  on  the  instructor’s  notes  instead  of  physically  connecting 
plugs,  turning  switches,  operating  a  bar  code  scanner,  etc.  Instruc¬ 
tors  commented  that  the  NMIS  makes  the  classroom  experience 
more  rewarding  for  instructors  because  it  has  a  positive  effect  on 
students’  attitudes  and  helps  the  students  to  stay  on  track. 
Instructors  said  unanimously  that  they  would  recommend  the  NMIS 
to  other  NJROTC  units  because  of  the  benefits  it  provides  for 
instructors  and  students. 
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DISCUSSION 

The  improvement  in  students’  scores  from  pre-test  to  post-test  sup¬ 
ports  the  assertion  that  the  NMIS  has  a  positive  effect  on  students’ 
academic  achievement.  However,  interpretation  of  the  test  scores  is 
subject  to  several  caveats.  On  one  hand,  scores  were  not  identified 
by  individual  (to  protect  students’  privacy),  and,  even  though  the  at¬ 
trition  in  the  matched  groups  was  comparable  during  the  period  of 
the  study,  the  outcomes  may  have  been  affected  by  disproportionate 
losses  of  students  who  had  different  levels  of  academic  performance. 
On  the  other  hand,  the  scores  may  understate  the  effect  of  the  NMIS. 
Naval  Science  1  students  had  to  learn  to  use  the  system,  and  they 
may  have  gained  proficiency  with  it  only  near  the  end  of  the  study 
period.  Further,  material  covered  during  the  first  ten  weeks  of  the 
courses  may  have  been  learned  relatively  easily;  had  the  study  been 
of  longer  duration,  the  courses  would  have  covered  more  material 
and,  perhaps,  more  challenging  material,  and  the  differentials  be¬ 
tween  the  groups’  average  scores  may  have  been  greater. 

Data  collected  on  the  students’  attitudes  about  courses  using  the 
NMIS  suggest  that  the  system  did,  indeed,  foster  higher  levels  of  aca¬ 
demic  achievement.  The  students’  comments  lend  support  to  the 
argument  that  the  factors  initially  posited  (i.e.,  multiple  representa¬ 
tions  of  material,  immediate  feedback,  labor-saving  tools)  were  in¬ 
strumental  to  the  outcome.  In  addition,  the  students’  comments 
suggest  that  other  factors  may  be  relevant  and  should  be  examined 
in  a  more  rigorous  study.  Specifically,  the  technology  itself  and  the 
active  interface  dialog  may  be  motivators.  Further,  the  discipline 
that  is  imposed  by  the  system  may  cause  students  to  do  work  that, 
otherwise,  they  would  avoid. 

Instructors  reported  that  the  system  has  a  positive  effect  on  their 
workload  in  terms  of  time  saved  and  enhanced  interaction  with  stu¬ 
dents.  Their  comments  suggest  that  the  NMIS  encourages  and  facili¬ 
tates  the  use  of  instructional  methods  that  are  well-regarded  by  stu¬ 
dents  and  instructors.  Some  of  these  methods  could  be  replicated,  at 
least  in  part,  by  additional  work  and  by  the  use  of  other  media,  e.g., 
transparencies.  However,  some  instructors  appear  to  be  daunted  by 
the  logistical  problems  of  using  multiple  media,  particularly  in  con¬ 
cert,  and  the  outcome  is  that,  without  the  NMIS,  lectures  typically 
include  few  visual  aids,  they  often  are  not  well-organized,  and,  from 
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the  students’  perspective,  they  often  are  dull,  hard  to  understand, 
and  not  engaging. 

In  its  present  form,  the  NMIS  is  used  to  deliver  instruction  in,  basi¬ 
cally,  a  traditional  manner.  We  believe  the  NMIS  presents  an  oppor¬ 
tunity  to  introduce  pedagogical  innovation  that  encourages  and  en¬ 
ables  students  to  become  more  actively  involved  in  their  education. 
We  encourage  CNET  to  consider  the  ways  they  might  promote  this 
type  of  change. 

This  study  did  not  investigate  the  implementation  of  the  system. 
However,  users’  comments  about  their  experiences  with  the  system, 
their  recommendations  for  on-going  support,  and  our  observations 
indicate  that  proper  implementation  and  on-going  support  are  criti¬ 
cal  to  the  system’s  success. 

CONCLUSIONS 

We  believe  the  findings  suggest  opportunities  for  CNET  that  go  be¬ 
yond  using  the  NMIS  solely  as  a  vehicle  to  deliver  instruction  via 
relatively  independent  systems  located  at  individual  schools.  The 
NMIS  enables  CNET  to  maintain  a  “virtual  presence”  at  NJROTC 
units.  It  presents  opportunities  for  pedagogical  innovation.  It  can  be 
used  to  enhance  the  interaction  of  NJROTC  units  with  the  school 
community.  Anecdotal  evidence  suggests  that  the  NMIS  can  help 
attract  more  and  better  students  to  the  NJROTC  program.  In  addi¬ 
tion,  NMIS  technology  enables  CNET  to  publish  and  distribute 
course  materials  electronically,  which  suggests  a  potential  for 
marked  cost-savings  in  the  administrative  and  logistical  costs  asso¬ 
ciated  with  traditional  textbooks  and  other  materials. 

The  findings  of  this  exploratory  study  cannot  be  generalized  to  other 
locations  because  the  sample  includes  only  two  schools,  which  may 
have  unique  characteristics;  the  sample  size  is  too  small  to  provide 
statistical  validity;  and  students  and  instructors  were  not  assigned 
randomly  to  groups,  consequently,  there  may  be  an  inherent  bias 
that  favors  one  group.  Even  so,  the  findings  are  informative  and,  we 
trust,  they  will  be  helpful  to  CNET. 

The  findings  indicate  that  a  more  rigorous  study  would  be  worth¬ 
while.  We  recommend  identifying  the  “critical  success  factors”  of  the 
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NMIS.  In  particular,  learning  more  about  the  training  and  technical 
support  needs  of  NMIS  users  (both  students  and  instructors)  would 
be  a  valuable  aid  to  designing  effective  implementation  procedures. 
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_ Chapter  One 

INTRODUCTION 


The  Navy  Junior  Reserve  Officers  Training  Corps  (NJROTC)  provides 
a  four-year  high  school  program  that  includes  an  academic  compo¬ 
nent  and  a  military  drill  leadership  experience.  Through  NJROTC 
course  work,  cadets  learn  about  world  politics  and  geography  and 
their  effect  on  world  affairs,  naval  history  and  traditions  including 
sea  and  space  science,  Navy  operations,  ships,  aircraft,  tactics,  career 
and  educational  planning,  and  personal  health  issues  (Sundt,  1991). 
In  the  main,  students  in  the  NJROTC  program  use  textbooks  and 
standardized  tests  provided  by  the  office  of  the  Chief  of  Naval  Edu¬ 
cation  and  Training  (CNET).  At  two  high  schools,  the  Naval  Science 
department  is  authorized  to  offer  courses  (such  as  American  Gov¬ 
ernment)  that  students  may  take  to  fulfill  regular  academic  require¬ 
ments;  in  those  courses,  students  use  the  school’s  standard  textbook. 

Approximately  ten  years  ago,  a  computer-based  NJROTC  multimedia 
instructional  system  (NMIS)  was  developed  to  enhance  the  delivery 
of  NJROTC  curricula.  The  NMIS  was  developed  by  Captain  John 
Nicholson  when  he  was  the  Naval  Science  Instructor  at  Sanger  High 
School,  Sanger,  California.  The  system  has  undergone  extensive  and 
relatively  continuous  refinement  since  its  inception,  and  it  has  been 
adopted  by  other  high  schools.  For  example,  it  was  implemented  in 
the  NJROTC  program  at  Lemoore  High  School,  Lemoore,  California, 
during  the  fall  term  of  school  year  1994-1995.  During  school  year 
1995-1996,  the  system  was  being  used  at  Lemoore  and  Sanger  to 
teach  Naval  Science  1,  Naval  Science  2,  American  History,  American 
Government,  and  Economics,  and,  at  Sanger  only,  Current  Events 
and  World  History. 
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In  this  study,  we  examined  the  effectiveness  of  the  NMIS  at  Lemoore 
and  Sanger  in  terms  of  three  broad  research  questions: 

1.  Does  the  system  have  a  positive  effect  on  students’  academic 
achievement? 

2.  What  are  the  students'  attitudes  toward  courses  using  the  system? 

3.  What  effect,  if  any,  does  the  system  have  on  instructors’  workload? 

The  data  collected  in  this  study  indicate  that  the  NMIS  has  a  positive 
effect.  However,  a  caveat  is  in  order.  It  would  not  be  appropriate  to 
conclude  that  similar  results  will  be  achieved  at  other  locations.  This 
was  an  exploratory  study  (Yin,  1989)  that  did  not  subject  the  asser¬ 
tion  to  a  formal  test  (e.g.,  it  did  not  use  randomly  selected  control 
groups  and  treatment  groups  or  a  sample  sufficiently  large  to  provide 
statistical  validity).  Even  with  this  caveat,  the  findings  of  this  study 
are  informative  and,  we  trust,  they  will  be  useful  to  CNET. 

A  description  of  the  system  is  provided  in  Chapter  Two  of  this  report. 
Chapter  Three  discusses  the  theoretical  foundations  of  the  three  re¬ 
search  questions.  Chapter  Four  describes  the  study  methodology, 
including  the  evaluation  design,  sample,  and  data  components. 
Chapter  Five  presents  the  findings  of  the  study  relative  to  the  three 
main  research  topics,  i.e.,  students’  achievement,  students  attitudes, 
and  instructors’  workload.  Chapter  Six  discusses  the  findings.  Chap¬ 
ter  Seven  presents  concluding  remarks,  which  include  our  opinion 
that  the  potential  of  the  NMIS  goes  beyond  its  current  applications  in 
ways  that  suggest  important  implications  for  CNET  and  the  NJROTC 
program. 


_ _ Chapter  Two 

DESCRIPTION  OF  THE  SYSTEM 


The  NMIS  is  designed  to  organize  students’  study  of  course  materi¬ 
als.  It  also  is  intended  to  help  instructors  integrate  supplementary 
material  (graphics,  video  clips,  text,  etc.)  into  their  lectures  and  into 
the  students’  reading  materials.  In  addition,  the  system  offers  in¬ 
structors  features  that  facilitate  assessment  and  record-keeping. 

The  NMIS  is  not  intended  to  replace  instructors.  Rather,  it  is  used  in¬ 
teractively  by  instructors  and  students  while  they  are  engaged  in  the 
exchange  of  information,  i.e.,  during  the  instructor’s  lecture  and  the 
discussion  period.  It  provides  a  guide  for  students  to  use  when  they 
are  taking  lecture  notes.  Students  also  use  it  to  read  the  course  ma¬ 
terials  and  take  notes  on  those  materials.  Further,  the  system  is  used 
to  administer  tests  of  the  students'  understanding  of  the  course  cur¬ 
riculum. 

The  NMIS  is  intended  to  pattern  users’  behavior  relative  to  a  set  of 
tasks.  To  appreciate  the  potential  effect  of  the  NMIS,  the  reader  must 
have  a  clear  understanding  of  the  system’s  components  and  its  func¬ 
tionality,  that  is,  the  manner  in  which  it  is  used  in  the  classroom. 
The  following  information  is  based  on  discussions  with  the  developer 
of  the  system,  on  discussions  with  instructors  and  students,  and  on 
our  observations  during  site  visits. 

THE  PHYSICAL  CONFIGURATION 

Lemoore  and  Sanger  each  have  one  NMIS  classroom.  It  contains  an 
instructor’s  workstation,  a  “home-theater’’-size  television  monitor, 
and  a  computer  for  each  student. 
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The  instructor’s  workstation  includes  a  computer  (i.e.,  computer 
unit,  monitor,  keyboard,  and  mouse),  videocassette  player,  laser  disc 
player,  and  a  switch  that  connects  the  output  from  either  the  video  or 
the  laser  disc  player  to  the  television  monitor.  The  instructor’s  ma¬ 
chine  has  supervisory  control  over  the  students’  machines,  that  is, 
instructors  can  download /up load  material  to/from  the  students 
machines,  and  they  can  lock  the  students’  keyboards  (e.g.,  to  prevent 
them  from  using  the  system  during  certain  class  activities).1 

The  NMIS  classroom  is  diagrammed  and  discussed  more  fully  in  Ap¬ 
pendix  A. 

Auxiliary  components  of  the  system,  which  are  located  in  an  adjacent 
room,  include  one  or  more  computers,  a  scanner,  and  a  printer.  In¬ 
structors  use  the  auxiliary  computers  to  develop  course  materials 
and  prepare  administrative  reports,  correspondence,  etc.  They  use 
the  scanner  to  digitize  tests  and  supplementary  course  materials 
such  as  newspaper  and  magazine  articles.  Students  use  the  auxiliary 
equipment  for  independent  research  and  to  prepare  materials  for 
NIROTC-related  activities;  for  example,  the  leader  of  the  drill  team  at 
one  school  uses  it  to  prepare  instructions  for  team  members. 

Software 

The  NMIS  is  constructed  from  off-the-shelf  software  packages  such 
as  Claris  Works  that  enable  applications  to  be  modified  via  user- 
accessible  features  (e.g.,  “routines,”  “procedures,”  and  “macros”). 
Expertise  with  information  systems  is  not  required  to  maintain  the 
system  (as  might  be  the  case,  for  example,  if  its  modules  had  been 
written  in  programming  code  such  as  “C”).  Instructors  require 
relatively  modest  technical  knowledge  to  develop  and/or  tailor 
course  material. 


lAn  instructor  said,  lock  the  students  keyboards  when  I  see  them  not  using  the 
system  for  an  assignment — or  when  they’re  typing  and  I’m  talking,  and  they  should 
not  be  taking  notes— for  example,  during  announcements,  when  I’m  giving  instruc¬ 
tions  for  a  test,  etc." 
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Links  to  External  Information  Sources 

Course  materials,  “Instructor’s  Notes,”  and  the  students’  files  are  the 
main  types  of  information  that  are  stored  within  the  NMIS;  they  are 
discussed  below.  In  addition,  the  system  provides  links  to  external 
sources  of  information. 

The  “Magazine  Data  Base”  was  complied  by  Captain  Nicholson  dur¬ 
ing  his  tenure  as  Naval  Science  Instructor  at  Sanger.  It  consists  of  a 
database  that  contains  citations  and  key  words  for  magazine  articles 
that  are  relevant  to  NJROTC  courses,  plus  hardcopies  of  the  maga¬ 
zines.  Students  use  the  database  and  hardcopy  collection  for  re¬ 
search  projects. 

One  computer  at  Sanger  is  linked  to  the  Internet.  At  Lemoore,  an 
Ethernet  line  is  being  extended  from  the  main  campus  communica¬ 
tion  node  to  the  NJROTC  building,  so  that  all  of  the  students’  ma¬ 
chines  will  have  access  to  the  Internet. 

Complementary  software  also  is  a  source  of  information.  For 
example,  reference  materials  are  available  on  compact  disks  at  both 
schools  and,  at  Sanger,  students  have  access  to  a  program  that 
provides  information  and  training  exercises  for  the  SAT. 

HOW  THE  NMIS  IS  USED  DURING  CLASS 

Typically,  the  NMIS  is  used  each  class  day  (excluding  military  drill 
days)  by  the  instructor  and  all  of  the  students  in  a  class.  Instructors 
can  tailor  the  manner  in  which  the  system  is  used  in  their  courses, 
and,  indeed,  we  observed  variations  in  its  use  among  the  classes  we 
visited.  During  a  typical  week,  instructors  use  the  system  to  inform 
students  of  the  week’s  schedule  and  assignments,  deliver  their  lec¬ 
tures  on  the  course  material,  stimulate  discussion,  and  administer 
tests.  Students  use  it  to  read  course  materials,  take  notes  on  the 
readings  and  lectures,  review  for  exams,  and  take  tests.  The  system 
also  may  be  used  for  special  assignments  such  as  essays  and  other 
projects. 
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Schedule  for  the  Week 

At  the  start  of  a  week,  the  instructor  displays  a  schedule  listing  the 
material  to  be  covered  and  the  assignments  for  the  week.  The 
schedule  is  displayed  on  the  television  screen  at  the  front  of  the 
classroom  and  on  each  student’s  monitor.  Most  students  copy  it 
(using  “cut  and  paste”  commands)  to  their  personal  files  for  future 
reference. 

Reading  the  Course  Materials  and  Taking  Notes 

Usually,  after  the  instructor  presents  the  objectives  for  the  day,  he 
asks  students  to  read  a  specified  portion  of  the  course  material.  They 
open  the  appropriate  files  on  their  computers,  and  course  material 
consisting  of  text  and  graphics  is  displayed  on  their  monitors.  Most 
of  the  material  was  prepared  by  third  parties  (e.g.,  textbooks  have 
been  digitized  and  input  to  NMIS).  However,  some  of  the  material  is 
prepared  by  the  instructor.  For  example,  one  instructor  inputs  arti¬ 
cles  from  the  morning’s  newspaper  (using  the  system  scanner) 
which,  he  states,  demonstrate  the  relevance  of  the  course  to  the  “real 
world.” 

Some  of  the  text  in  the  course  material  is  underlined  (indicating  a 
hypertext  link).  When  the  text  is  “clicked,”  a  pop-up  window  appears 
that  contains  graphics  (e.g.,  pictures,  maps,  and  graphs)  or  a  defini¬ 
tion  of  the  underlined  word  or  phrase. 

Students’  monitors  display  two  vertical  windows  while  they  are 
reading  the  course  material.  Course  material  appears  in  the  left  win¬ 
dow.  Students  read  it  at  their  own  pace.  As  they  read,  they  write 
notes  in  their  personal  “reading  notes”  file,  which  appears  in  the 
right  window.  The  software  organizes  their  notes  in  standard  outline 
format  (i.e.,  I,  A,  1,  a).  Students  can  color  the  text  in  their  notes  to 
highlight  important  items. 

While  students  are  reading  the  course  material,  the  instructor  can 
attend  to  other  tasks,  e.g.,  take  attendance,  do  other  administrative 
work,  assist  students  individually,  and  walk  around  the  classroom  to 
view  the  work  being  done  by  the  class  (he  also  can  access  students 
notes  from  his  computer  to  review  their  work  in  detail). 
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Listening  to  the  Lecture  and  Taking  Notes 

As  the  instructor  lectures,  “Instructor’s  Notes”2  are  displayed  on  the 
large  television  screen  at  the  front  of  the  classroom.  The  “Notes"  are 
comparable  to  the  transparencies  and  slides  that  are  used  for  lec¬ 
tures  in  classes  at  graduate  schools  of  business  and  for  presentations 
in  business,  military,  and  other  organizations.  Instructors  use  a 
mouse  to  control  the  pacing  and  the  display  of  material  on  the  tele¬ 
vision  screen.  They  can  display  entire  “screens”  or  they  can  choose 
to  display  one  section  of  the  screen  at  a  time,  rather  as  a  briefer  who 
is  using  transparencies  might  slide  an  opaque  sheet  of  paper  down  a 
transparency  to  reveal  the  next  point  to  be  addressed. 

Students  call  up  an  abbreviated  version  of  the  Instructor’s  Notes  to 
the  left  window  of  their  screens,  and  copy  it  (using  “cut  and  paste” 
commands)  to  their  personal  “lecture  notes"  file  in  the  right  window. 
The  students’  version  contains  major  and  minor  headings,  but  not 
the  detailed  information  under  those  headings  that  is  displayed  on 
the  television  monitor.  Students  are  encouraged  to  annotate  their 
copy  during  the  lecture.  In  particular,  they  are  encouraged  to  take 
notes  on  “foot  stampers, ”  i.e.,  issues  stressed  by  the  instructor  “that 
you  are  likely  to  see  again”  (in  an  exam).  In  a  student’s  file,  outline 
text  is  colored  red  and  blue,  and  the  student’s  notes  are  colored 
black.  This  convention  enables  students  and  instructors  to  identify 
quickly  a  student’s  annotations. 

At  the  instructor’s  discretion,  the  course  materials  (e.g.,  the  textbook) 
may  be  available  in  the  left  window.  Consequently,  students  can 
have  simultaneous  access  to  up  to  three  different  sources  of  infor¬ 
mation:  (1)  the  Instructor’s  Notes  on  the  television  screen  at  the 
front  of  the  room,  (2)  their  personal  notes  in  the  right  window  of 
their  screens,  and  (3)  the  text  and  other  course  materials  in  the  left 
window. 

Video  clips  from  the  CNET  laser  disk  are  interwoven  seamlessly  into 
the  instructor’s  lecture.  Instructors  install  “buttons”  in  their  Notes  at 
places  where  they  want  to  illustrate  their  comments  with  items  from 
the  laser  disk.  During  their  lecture,  they  click  a  button  to  display  an 
item.  The  buttons  are  linked  to  laser  disk  bar-codes.  This  procedure 


2 In  some  courses,  the  outline  was  scanned  from  the  NJROTC  Instructor  Guide. 
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simplifies  the  former  practice  which  required  instructors  to  locate 
the  appropriate  bar  codes  during  their  lecture,  and  then  scan  them 
with  a  reading  device. 

Instructors  have  installed  links  from  NMIS  to  other  software  that  they 
use  during  their  lectures.  For  example,  they  use  map  software  which 
displays  the  Earth  in  color  on  the  television  screen.  The  software  ro¬ 
tates  the  globe,  focuses  on  a  large  region  (e.g.,  Europe  and  the  Middle 
East),  narrows  the  focus  to  a  smaller  region  (e.g.,  Italy,  Greece  and 
Turkey),  further  narrows  to  a  country  (e.g.,  Italy);  and,  finally,  nar¬ 
rows  to  an  area  within  the  country  (e.g.,  The  Piedmont). 

Instructors  can  walk  around  the  room  as  they  lecture  and  use  a 
cordless  mouse  to  control  the  pacing  and  display  of  the  Instructor’s 
Notes.  Among  the  benefits  of  this  practice:  Instructors  can  observe 
students’  note-taking  activity.  Of  course,  instructors  also  can  review 
the  students’  lecture  notes  in  detail  by  accessing  the  students’  files 
via  the  instructor’s  computer  (just  as  they  can  access  and  review 
students’  notes  on  the  readings). 

Classroom  Discussion 

During  and/or  after  their  lectures,  instructors  engage  students  in  dia¬ 
log  about  the  subject  matter.  Sections  of  Instructor’s  Notes  and 
items  from  the  CNET  laser  disk  can  be  revisited  easily  by  the  instruc¬ 
tor,  when  appropriate. 

NMIS  provides  a  novel  method  for  reviewing  course  material  and 
eliciting  discussion  among  the  students.  At  the  touch  of  a  mouse 
button,  the  instructor  displays  a  question  on  the  television  monitor 
at  the  front  of  the  room.  The  system  generates  a  random  number 
between  1  and  28  inclusive,  displays  it  in  the  corner  of  the  screen, 
and  begins  to  play  the  theme  music  from  the  television  quiz  show 
“Jeopardy.”  The  number  designates  a  computer  station.  When  the 
music  stops,  the  student  who  is  seated  at  the  designated  computer 
gives  his  or  her  answer.  Then  the  instructor  clicks  his  mouse  to  re¬ 
veal  the  correct  answer  on  the  television  screen. 

Some  class  sessions,  particularly  those  immediately  before  exams, 
are  used  to  review  course  material  and  lectures.  In  some  classes, 
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students  are  able  to  print  a  hard  copy  of  their  notes  on  the  readings 
and  lectures,  and  review  them  outside  the  classroom. 

Assignments,  Essays,  and  Special  Projects 

“Homework”  is  something  of  a  misnomer  because  most  of  the  as¬ 
signments  are  done  during  the  class  period.  (Homework  in  Naval 
Science  1,  generally,  consists  of  the  questions  in  the  text  and  vo¬ 
cabulary  exercises.)  However,  some  students  do  their  “homework” 
during  lunch  and  after  school. 

The  use  of  essays  and  special  projects  varies  by  course  and  by  in¬ 
structor.  Students  can  be  asked  to  write  essays  using  their  notes  on 
the  course  material  and/or  on  the  lectures.  Their  computer  screen 
displays  two  windows.  The  student’s  notes  appear  in  the  left  win¬ 
dow.  The  essay  is  written  in  the  right  window. 

Some  instructors  call  up  students’  homework  to  the  instructor’s 
computer,  copy  each  student’s  work  to  a  new  file,  comment  on  the 
work  (in  the  new  file)  using  text  and  graphics  provided  by  the  NMIS 
editor,  and  return  the  annotated  homework  to  the  students.  This 
procedure  (putting  the  instructor-annotated  homework  in  a  separate 
file)  preserves  the  unmarked  files,  thus  documenting  the  students’ 
progress  during  the  course. 

Examinations 

Examinations  are  administered  using  a  software  package  (“LXR 
Test”)  that  has  been  integrated  transparently  into  the  NMIS.  The 
package  prepares  tests  in  various  formats,  e.g.,  multiple  choice, 
true/false,  matching,  completion,  numeric,  and  open-ended.  It  also 
allows  graphics  to  be  used.  For  example,  one  instructor  scanned  a 
map,  replaced  the  country  names  with  “A,”  “B,”  etc.,  then  asked  stu¬ 
dents  to  name  “Country  B.” 

When  students  take  an  exam  at  their  computers,  they  are  informed 
whether  they  are  correct  each  time  they  answer  a  question  (after  they 
“mark”  or  finalize  their  answers).  If  they  are  incorrect,  the  correct 
answer  is  shown. 


10  Evaluation  of  the  NJROTC  Multimedia  Instructional  System 


During  the  exam,  students  are  informed  of  the  number  of  unan¬ 
swered  questions  and  the  amount  of  time  remaining  in  the  test  pe¬ 
riod.  At  the  end  of  the  exam,  they  are  given  a  summary  report  of 
their  performance.  Each  student’s  answers  are  available  to  the  in¬ 
structor’s  computer,  and  reports  are  prepared  for  him  showing 
grades  by  student  (as  raw  scores,  percentages,  or  letter  grades),  class 
rankings,  class  performance  by  question,  etc. 


_ Chapter  Three 

THE  RESEARCH  QUESTIONS 


Learning  theory  suggests  that  the  NMIS  has  numerous  distinctive 
features  that  may  enrich  students’  classroom  experience  and  help 
them  assimilate  the  curriculum  better  than  traditional  methods  of 
instruction.  For  example,  lectures  seamlessly  incorporate  graphical 
materials  from  the  CNET  laser  disk  and  other  sources,  and  reading 
materials  are  presented  with  hypertext  links  to  pictures  and  a  glos¬ 
sary.  According  to  learning  theory,  multiple  representations  of  ma¬ 
terial  are  likely  to  promote  retention  on  the  part  of  students  (Kaput, 
1989). 

In  addition,  immediate  feedback  during  exams  provides  positive  re¬ 
inforcement  that,  as  explained  by  behavioral  theory  and  extensions 
to  it  (cf.  Bandura,  1982),  plays  an  important  role  in  the  acquisition  of 
new  behaviors.  Further,  the  system  organizes  students’  and  instruc¬ 
tors’  activities  in  ways  that  may  facilitate  the  delivery,  comprehen¬ 
sion,  and  retention  of  course  material. 

These  considerations  lead  to  the  first  broad  research  question: 

1.  Do  students  who  are  enrolled  in  classes  that  use  the  NMIS  achieve 
higher  levels  of  academic  performance  than  do  students  in  classes 
that  are  taught  with  traditional  methods  of  instruction? 

The  tools  embedded  within  the  NMIS  may  enable  students  to  work 
more  effectively  and  instructors  to  work  more  productively  relative 
to  traditional  methods  of  instruction.  However,  information  systems 
incur  costs  as  well  as  benefits  (Eason,  1984;  Noble,  1984),  and  both 
the  costs  and  benefits  can  be  distributed  disproportionately  among 


it 
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groups  of  users  (Grudin,  1988).  Users’  perceptions  of  their  personal 
costs  and  benefits  are  likely  to  affect  their  motivation  to  use  a  system 
(Devin,  1994). 

Consequently,  it  is  salient  to  learn  whether  students  are  engaged  by 
the  system  and  whether  they  perceive  it,  on  balance,  to  improve  the 
learning  process  for  them. 

2.  What  are  the  students’  attitudes  about  NJROTC  courses  that  use 
the  NMIS  relative  to  courses  that  are  taught  in  the  traditional 
manner? 

Similarly,  it  is  appropriate  to  ask  instructors  whether  the  cost/benefit 
ratio  is  equitable  for  them,  and  whether  “productivity”  gains,  if  any, 
have  positive  net  outcomes. 

3.  What  is  the  perceived  effect  of  the  system  on  instructors  work¬ 
load? 

The  three  research  questions  are  used  as  devices  to  explore,  orga¬ 
nize,  and  focus  the  constellation  of  wide-ranging  issues  that  are  en¬ 
compassed  by  this  study. 


Chapter  Four 

METHODOLOGY 


This  study  compares  students  in  classes  using  the  NMIS  with  stu¬ 
dents  in  classes  not  using  the  system.  We  were  unable  to  randomly 
assign  students  to  treatment  and  control  groups  because  of  logistical 
and  time  constraints.  However,  before  the  start  of  the  study,  stu¬ 
dents  were  assigned  to  classrooms  based  on  standard  school 
scheduling  procedures.  Some  of  the  classrooms  were  equipped  with 
the  NMIS  while  other  classrooms  were  not.  This  study  makes  use  of 
these  naturally  occurring  groups.1 

We  examined  the  use  of  the  NMIS  at  two  high  schools,  Lemoore  and 
Sanger,  during  the  fall  1995  school  term.  The  principals  of  the 
schools  notified  parents  of  the  study  and  requested  them  to  inform 
the  principal  if  they  did  not  want  their  child  to  participate  in  it.  We 
were  advised  that  one  parent  at  Lemoore  and  no  parents  at  Sanger 
refused  participation  for  their  child. 

THE  SAMPLE 

The  sample  consists  of  more  than  250  students  who  were  enrolled  in 
three  courses:  Naval  Science  1  (NS  1)  at  Lemoore,  NS  1  at  Sanger, 
and  American  Government  at  Sanger.  These  courses  were  selected 
because  some  sections  of  each  course  were  taught  using  the  NMIS, 


hn  this  report,  terms  such  as  the  “NMIS  group,”  “NMIS  sections,”  and  “NMIS  classes” 
refer  to  students  who  used  the  system.  Similarly,  terms  such  as  the  "Traditional 
group,”  "Traditional  sections,”  and  “Traditional  classes”  refer  to  students  who  were 
taught  with  traditional  methods  of  instruction. 
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and  other  sections  were  instructed  with  traditional  methods.2  The 
sample  is  summarized  in  Table  4.1,  which  shows  by  course,  section 
(class  period),  and  method  of  instruction,  the  number  of  students 
who  took  the  pre-test  (discussed  below)  at  the  beginning  of  this 
study. 


Table  4.1 

Participation  in  the  Study 


Class  Number  of  Students 


Period 

NMIS  Traditional 

Instructor 

Naval  Science  1,  Lemoore 

2 

18 

Mr.  A 

3 

19 

Mr.  A 

4 

18 

Mr.  A 

7 

26 

Mr.  A 

Naval  Science  1,  Sanger 

2 

26 

Mr.  B 

5 

22 

Mr.  B 

1 

29 

Mr.  C 

2 

27 

Mr.  D 

American  Government,  Sanger 

6 

30 

Mr.  B 

4 

29 

Mr.  E 

5 

35 

Mr.  E 

Total 

133 

146 

Naval  Science  1,  Lemoore  High  School 

The  Naval  Science  1  course  at  Lemoore  had  four  sections.  All  of  the 
sections  were  taught  by  the  same  instructor.  Three  of  the  sections 
were  scheduled  to  be  given  in  the  NMIS-equipped  classroom.  How¬ 
ever,  before  the  school  year  started,  more  students  were  assigned  to 
the  fourth  section  than  the  NMIS  classroom  could  accommodate,  so 


2We  also  believed  the  data  collected  from  the  NS  1  courses  would  be  particularly  in¬ 
formative  because  these  students  would  be  experiencing  the  NMIS  for  the  first  time 
(unlike  students  in  advanced  courses  who  were  likely  to  have  used  the  system  previ¬ 
ously),  and  they  would  draw  upon  recent  experience  with  the  system  to  discuss  the 
ease  of  learning  to  use  it  and  their  initial  impressions  of  it. 
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this  section  was  scheduled  to  be  held  in  a  typical  classroom  where 
traditional  methods  of  instruction  would  be  used. 

Some  of  the  students  dropped  the  course  during  the  school  term  (the 
rate  of  attrition  was  comparable  in  all  of  the  sections).  The  section 
being  taught  with  traditional  methods  became  small  enough  to  move 
to  the  NMIS  classroom.  However,  this  change  was  deferred  until  the 
end  of  the  formal  period  of  the  study,  that  is,  until  after  the  post-test 
was  administered. 

Naval  Science  1,  Sanger  High  School 

Sanger’s  Naval  Science  1  course  had  eight  sections.  Two  sections 
were  taught  in  the  NMIS-equipped  classroom.  Six  sections  were 
conducted  in  a  typical  classroom  using  traditional  methods  of  in¬ 
struction  because  they  were  scheduled  during  periods  when  the 
NMIS-equipped  classroom  was  occupied  by  other  classes.  Four  sec¬ 
tions  were  selected  to  participate  in  this  study:  the  two  sections 
using  the  NMIS,  and  two  of  the  sections  using  traditional  methods  of 
instruction.  The  two  NMIS  sections  were  taught  by  the  same  instruc¬ 
tor.  The  two  Traditional  sections  were  taught  by  two  other  instruc¬ 
tors. 

At  the  start  of  the  school  year,  121  students  were  officially  enrolled  in 
these  four  NS  1  sections  (Table  4.1  shows  the  number  of  students 
who  took  the  pre-test).  Approximately  47  percent  of  the  students 
were  in  the  NMIS  group  and  53  percent  were  in  the  Traditional 
group.  These  percentages  held  relatively  constant  during  the  period 
of  the  study.  We  are  advised  that  no  students  transferred  between 
the  sections  of  the  course  after  the  pre-test  was  administered. 

American  Government,  Sanger  High  School 

The  American  Government  course  at  Sanger  consisted  of  eleven  sec¬ 
tions.  An  NJROTC  instructor  taught  one  college  prep  section  using 
the  NMIS.  Teachers  in  the  Social  Science  Department  used 
traditional  methods  of  instruction  to  teach  the  other  ten  sections,  of 
which  four  were  college  prep  sections  and  six  were  “regular” 


16  Evaluation  of  the  NJROTC  Multimedia  Instructional  System 


sections.  The  same  textbook  was  used  in  all  five  college  prep 
sections  of  the  course. 

The  principal  of  the  high  school  informed  us  that  students  were  dis¬ 
tributed  randomly  among  the  sections  of  the  course — with  the  fol¬ 
lowing  exceptions:  (1)  college  prep  students  were  separated  from 
“regular”  students,  (2)  two  of  the  college  prep  sections  were 
“remedial,”  i.e.,  for  students  with  reading  problems,  and  (3)  NJROTC 
determined  who  would  be  admitted  to  its  section  of  the  course. 

The  Naval  Science  Instructor  informed  us  that,  typically,  more  stu¬ 
dents  applied  for  the  NJROTC  section  of  the  course  than  it  could  ac¬ 
commodate.  On  August  3,  1995,  65  students  had  applied  for  the  30 
slots  in  the  course,  and  more  students  were  expected  to  apply  before 
the  school  year  began.  Usually,  NJROTC  accepted  only  those  stu¬ 
dents  (1)  who  had  been  in  NJROTC  for  at  least  three  years  and  (2) 
who  had  “demonstrated  leadership,”  that  is,  who  were  suitable  to 
serve  as  NJROTC  cadet  officers  and  leaders. 

The  NJROTC  section  of  the  American  Government  course  and  two  of 
the  "non-remedial”  college  prep  sections  were  selected  to  participate 
in  this  study.  The  instructors  agreed  to  cover  the  same  chapters  of 
the  textbook  during  the  period  of  the  study.  The  three  sections  had 
comparable  numbers  of  students;  approximately  30  percent  of  the 
students  were  in  the  NMIS  group. 

Students  did  not  transfer  between  the  NMIS  section  and  the  Tradi¬ 
tional  sections  of  the  course. 

STUDENTS’  CHARACTERISTICS 

We  collected  demographic  data  from  the  schools’  records  to  examine 
the  comparability  of  the  students  in  the  NMIS  and  Traditional 
groups.  The  data  for  students  in  the  NS  1  course  are  summarized  in 
Table  4.2  by  method  of  instruction;  they  describe  the  students’  grade 
level,  gender,  ethnicity,  standardized  test  scores,  and  participation  in 
the  free  or  reduced-price  lunch  program. 
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Naval  Science  1,  Lemoore  High  School 

As  reported  in  Table  4.2,  the  average  CTBS  scores3  of  NS  1  students  in 
the  Traditional  group  at  Lemoore  were  higher  than  the  average 
scores  of  the  students  in  the  NMIS  group  by  10,  11,  and  8  points, 
respectively,  for  reading,  language,  and  math.  The  differences 
between  the  groups  are  not  statistically  significant.  However,  the 
data  indicate  that,  on  average,  students  in  the  Traditional  group  were 
more  proficient  in  reading,  language,  and  math  skills  than  students 
in  the  NMIS  group. 


Table  4.2 

Demographic  Data  for  NS  1  Course  by  Method  of  Instruction 


NS  1 

Lemoore3 

NS  1 

Sangerb 

NMIS 

Traditional 

NMIS 

Traditional 

Scores 

Reading 

772 

782 

745 

754 

Language 

756 

767 

748 

742 

Math 

779 

787 

761 

757 

Grade  level 

9th 

71% 

83% 

58% 

47% 

10th 

27% 

13% 

32% 

47% 

11th 

1% 

0 

5% 

3% 

12th 

1% 

4% 

5% 

3% 

Gender 

Male 

44% 

42% 

65% 

59% 

Female 

56% 

58% 

35% 

41% 

Free/reduced  lunch 

18% 

17% 

23% 

13% 

Ethnicity 

Hispanic 

15% 

4% 

65% 

53% 

All  other 

85% 

96% 

35% 

47% 

aData  pertain  to  the  95  students  who  were  officially  enrolled  in  all  sections 
of  the  NS  1  course  when  the  post-test  was  administered. 
bData  pertain  to  the  121  students  who  were  officially  enrolled  at  the  start 
of  the  school  year  in  the  four  NS  1  sections  that  participated  in  this  study. 


3 Lemoore  scores  are  from  the  Spring  1996  Comprehensive  Tests  of  Basic  Skills  (CTBS), 
which  included  “reading"  (vocabulary  and  comprehension),  “language”  (mechanics 
and  expression),  and  “math”  (computation,  and  concepts  and  applications). 
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The  NMIS  group  had  proportionately  fewer  freshmen  and  propor¬ 
tionately  more  sophomores  than  the  Traditional  group.  The  distri¬ 
bution  of  male  and  female  students  was  approximately  equal  in  the 
two  groups.  The  two  groups  are  relatively  comparable  in  terms  of  the 
percentage  of  students  participating  in  the  federally  funded  free  or 
reduced-price  lunch  program.  (We  use  this  measure  as  a  surrogate 
for  the  students’  socioeconomic  status  (SES) .)  The  NMIS  group  had 
a  larger  proportion  of  Hispanic  students  than  the  Traditional  group. 

Naval  Science  1,  Sanger  High  School 

Table  4.2  shows  that,  on  average,  students  in  the  Traditional  group 
scored  nine  points  higher  than  did  students  in  the  NMIS  group  on 
the  CAT/5  reading  module,  six  points  lower  than  the  NMIS  group  on 
language,  and  four  points  lower  on  math.4  The  differences  are  not 
statistically  significant.  However,  the  data  indicate  that,  on  average, 
students  in  the  Traditional  group  were  more  proficient  in  reading 
skills  than  were  students  in  the  NMIS  group. 

The  NMIS  group  had  proportionally  more  freshmen  and  fewer 
sophomores  than  did  the  Traditional  group.  The  NMIS  group  had  a 
somewhat  larger  proportion  of  female  students  than  the  Traditional 
group.  In  comparison  with  students  in  the  Traditional  group,  almost 
twice  as  many  students  in  the  NMIS  group  participated  in  the  re¬ 
duced-price  or  free  lunch  program.5 

More  than  50  percent  of  the  students  in  the  four  sections  were  His¬ 
panic.  This  statistic  is  representative  of  the  student  population  at 


4Sanger  scores  are  from  the  Fall  1995  California  Achievement  Tests  (CAT/5),  which 
included  "reading”  (comprehension  only),  “language"  (mechanics  and  expression), 
and  “math”  (computation,  and  concepts  and  applications). 

5Even  though  the  NS  1  groups  at  Sanger  are  not  comparable  in  terms  of  SES  (as  indi¬ 
cated  by  participation  in  the  free  or  reduced-price  lunch  program),  students  com¬ 
ments  during  the  focus  groups  suggest  that  economic  advantage/disadvantage  was 
not  directly  relevant  to  their  access  to  systems  technology.  In  general,  Sanger  students 
reported  that  they  gained  basic  experience  with  information  systems  by  using 
computers  at  their  elementary  schools  and/or  junior  high  schools.  They  also  reported 
that  while  they  were  in  grade  school,  they  could  borrow  laptop  computers  from  the 
school  library  and  take  them  home,  in  the  same  manner  that  they  borrowed  books. 
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Sanger.  However,  Table  4.2  shows  that  approximately  65  percent  of 
the  students  in  the  NMIS  group  were  Hispanic  (a  ratio  of  1.8  to  stu¬ 
dents  in  all  other  ethnic  groups),  whereas  53  percent  of  the  students 
in  the  Traditional  group  were  Hispanic  (a  ratio  of  1.1). 

The  distribution  of  students  by  ethnicity  is  relevant  because,  on  av¬ 
erage,  Hispanic  students’  CAT/5  scores  were  substantially  lower  than 
the  scores  of  students  in  all  other  ethnic  groups.  Relative  to  other 
students’  scores,  the  scores  of  Hispanic  students  were,  on  average,  35 
points  lower  in  reading,  36  points  lower  in  language,  and  35  points 
lower  in  math.  These  differences  are  statistically  significant. 

The  data  suggest  that  the  NMIS  group,  on  average,  had  a  lower 
CAT/5  reading  score  than  the  Traditional  group  (745  versus  754)  be¬ 
cause  the  NMIS  group  had  a  larger  proportion  of  Hispanic  students 
(65  percent  versus  53  percent),  and  the  Hispanic  students’  scores,  on 
average,  are  lower  than  the  average  scores  of  students  in  other  ethnic 
groups. 

American  Government,  Sanger  High  School 

All  of  the  students  in  the  three  participating  sections  of  the  American 
Government  course  were  seniors.  As  reported  in  Table  4.3,  relative 
to  the  Traditional  group,  the  NMIS  group  had  proportionately  more 
female  than  male  students  and  proportionately  more  Hispanic 
students  than  students  of  other  ethnicities.6 

DATA  COLLECTION 

Evaluation  data  consist  of  scores  on  a  pre-test  and  post-test,  stu¬ 
dents’  comments  during  focus  groups  with  users  and  non-users  of 
the  system,  interviews  with  instructors,  and  observations  of  classes. 


6Demographic  data  were  collected  from  the  schools’  records  after  the  post-tests  were 
administered.  On  reviewing  the  post-test  scores  in  the  American  Government  course, 
we  determined  that  our  understanding  of  them  would  not  be  illuminated  either  by  the 
students’  scores  on  the  CAT  exam  or  by  their  participation  in  the  free  or  reduced-price 
lunch  program,  and  we  elected  not  to  spend  the  resources  that  would  have  been  re¬ 
quired  to  collect  that  data. 
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Table  4.3 

Demographic  Data  for  the  American  Government 
Course  by  Group 


_ NMIS _ Traditional 

Grade  Level 


12th 

100% 

100% 

Gender 

Male 

43% 

50% 

Female 

57% 

50% 

Ethnicity 

Hispanic 

37% 

25% 

All  other 

63% 

75% 

NOTE:  Data  pertain  to  the  94  students  who  took  the 
pre-test. 


The  Pre-Test  and  Post-Test 

Students’  academic  achievement,  i.e.,  the  focus  of  the  first  research 
question,  was  measured  by  their  scores  on  a  test  covering  material 
that  the  instructors  planned  to  include  in  the  course  during  the  pe¬ 
riod  of  this  study.  The  NS  1  test  consisted  of  questions  taken  from 
CNET’s  test  battery.  The  American  Government  test  consisted  of 
questions  selected  from  a  test  battery  prepared  by  the  publisher  of 
the  textbook  that  was  used  by  the  three  sections  of  the  American 
Government  course  that  participated  in  this  study. 

The  pre-test  was  administered  to  the  students  at  the  start  of  the  term 
to  measure  their  knowledge  of  the  subject  matter  before  instruction 
began.  The  post-test  was  given  near  the  end  of  the  term  to  measure 
their  knowledge  of  the  subject  matter  after,  approximately,  10  weeks 
of  instruction.  The  tests  were  prepared,  administered,  and  graded 
by  RAND  personnel. 

Focus  Groups 

Students’  attitudes  about  NJROTC  courses  using  the  system,  i.e.,  the 
central  issue  of  the  second  research  question,  were  addressed  during 
focus  group  sessions.  Eleven  focus  groups,  with  124  students  in  total, 
were  conducted  at  Lemoore  and  Sanger. 
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At  Lemoore,  we  met  with  students  in  all  of  the  NJROTC  courses — in¬ 
cluding  courses  that  were  not  participating  in  the  study.  Students  in 
the  non-participating  courses  were  using  the  NMIS  currently,  and 
most  of  them  had  used  it  during  the  previous  academic  year.  We  met 
with  them  to  collect  data  regarding  the  perceptions  of  relatively  well- 
experienced  users  of  the  system. 

Students  in  the  Traditional  group  at  Lemoore  had  begun  to  use  the 
NMIS  shortly  after  the  post-test  was  administered.  When  we  met 
with  them,  they  had  been  using  the  NMIS  for  approximately  four 
weeks.  We  were  particularly  interested  in  their  initial  reaction  to  the 
system  and  their  perceptions  of  it  relative  to  traditional  methods  of 
instruction.  We  met  again  with  this  group  approximately  three 
months  later  to  determine  whether  their  initial  views  of  the  system 
had  changed  as  they  gained  familiarity  with  it. 

At  Sanger,  we  met  with  NJROTC  students  in  (1)  all  three  of  the  partic¬ 
ipating  groups  that  used  the  NMIS,  (2)  one  of  the  participating  sec¬ 
tions  that  was  using  traditional  methods  of  instruction,  and  (3)  a 
group  composed  of  students  in  advanced  NJROTC  courses  who  had 
used  the  NMIS  previously  during  one  or  more  years. 

Approximately  half  of  the  students  in  each  class  section  were  ran¬ 
domly  chosen  to  attend  the  focus  groups.  Each  focus  group  con¬ 
sisted  of  approximately  12  students.  The  participants  adjourned  to  a 
room  outside  the  classroom  to  discuss  the  system  with  RAND  per¬ 
sonnel.  Instructors  were  not  present  during  the  focus  group  ses¬ 
sions. 

In  the  main,  topics  discussed  with  users  included:  differences  be¬ 
tween  classes  using  the  NMIS  and  classes  using  traditional  methods 
of  instruction;  experience  using  the  system,  including  the  process  of 
learning  to  use  it;  effectiveness  of  specific  components  of  the  system; 
uses  of  the  system  outside  of  NJROTC  class;  and  recommendations 
for  improving  the  NMIS.  We  asked  students  who  had  seen  the  sys¬ 
tem,  but  who  had  not  used  it  in  class,  about  their  perceptions  of 
classes  taught  using  the  NMIS  versus  those  taught  with  traditional 
methods  of  instruction. 
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Interviews  of  Instructors 

The  constellation  of  issues  relating  to  the  third  research  question  was 
discussed  during  interviews  with  the  instructors.  All  of  the  instruc¬ 
tors  who  used  the  system  were  interviewed.  Their  comments  cov¬ 
ered  (1)  their  views  regarding  the  effect  of  the  system  on  their  work¬ 
load,  pedagogy,  and  their  interaction  with  their  students;  (2)  their 
estimates  of  the  amount  of  training  instructors  need  to  use  the  sys¬ 
tem;  and  (3)  their  overall  assessments  of  the  system,  i.e.,  whether 
they  would  recommend  it  to  other  NJROTC  units. 

Observation  of  Classes 

We  visited  classes  to  collect  information  that  would  supplement  and 
illuminate  the  data  that  were  collected  during  the  tests,  focus  groups, 
and  interviews.  We  observed  all  of  the  class  sections  at  Lemoore,  in¬ 
cluding  sections  of  advanced  NJROTC  courses  where  most  of  the 
students  had  used  the  system  in  one  or  more  courses  previously.  We 
included  advanced  courses  to  examine  whether  students’  interaction 
with  the  system  changed  as  they  gained  experience  with  it,  and 
whether  there  were  differences  in  the  use  of  the  system  among  the 
instructors.  Similarly,  we  observed  approximately  half  of  the 
NJROTC  class  sections  at  Sanger  and  one  of  the  Social  Science  De¬ 
partment  courses  that  was  participating  in  this  study. 


Chapter  Five 

FINDINGS 


The  findings  of  this  study  are  organized  by  the  three  research  ques¬ 
tions  regarding  (1)  students’  academic  achievement,  (2)  students’ 
views  regarding  the  system,  and  (3)  instructors’  perceptions  of  the 
efficacy  of  the  system  relative  to  their  workload.  This  chapter  also 
includes  data  about  the  integration  of  the  NMIS  into  the  larger 
school  culture. 

STUDENTS’  ACADEMIC  ACHIEVEMENT 

As  reported  in  Table  5.1,  the  average  scores  of  students  using  the 
NMIS  increased  more  from  pre-test  to  post-test  than  did  the  average 
scores  of  students  who  were  taught  with  traditional  methods  of  in¬ 
struction.  We  focus  on  the  increase  in  scores  because  two  of  the 
matched  groups  (NS  1  at  Lemoore  and  American  Government  at 

Table  5.1 

Average  Scores  on  Pre-Test  and  Post-Test 


Method 

Pre-Test 

Post-Test 

Change 

NS  1,  Lemoore 

NMIS 

31.7 

42.1 

+10.4 

32.8% 

Traditional 

37.9 

43.0 

+5.1 

13.5% 

NS  1,  Sanger 

NMIS 

33.7 

41.5 

+7.8 

23.1% 

Traditional 

33.4 

38.7 

+5.3 

15.9% 

American 

NMIS 

17.2 

22.3 

+5.1 

29.7% 

Government, 

Sanger 

Traditional 

19.2 

17.8 

-1.4 

-7.3% 
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Sanger)  did  not  have  comparable  familiarity  with  the  course  material 
at  the  start  of  the  study. 

Brief  descriptions  of  the  data  follow.  More  detailed  descriptions  can 
be  found  in  the  appendices.  Note  that  the  same  tests  were  used  in 
the  NS  1  courses  at  Sanger  and  Lemoore,  but  the  scores  should  not 
be  compared  directly  because  the  two  courses  covered  somewhat 
different  material. 

Naval  Science  1,  Lemoore  High  School 

The  NMIS  group  in  NS  1  at  Lemoore  scored  6.2  points  less  on  the 
pre-test  than  did  the  Traditional  group,  and  the  difference  is  highly 
significant  (p  =  0.0002).  However,  the  difference  in  the  groups  post¬ 
test  scores  is  not  statistically  significant  (p  =  0.6390).  In  other  words, 
the  data  indicate  there  is  a  zero  probability  that  students  in  the  two 
groups  had  the  same  knowledge  of  the  course  material  when  the 
study  began,  but  their  knowledge  of  the  material  was  not  signifi¬ 
cantly  different  when  the  study  ended. 

The  average  score  of  the  NMIS  group  increased  32.8  percent  (i.e.,  up 
10.4  points,  from  31.7  to  42.1)  whereas  the  average  score  of  the  Tradi¬ 
tional  group  increased  13.5  percent  (i.e.,  up  5.1  points,  from  37.9  to 
43.0).  The  6.2  point  “advantage”  of  the  Traditional  group  on  the  pre¬ 
test  (37.9  versus  31.7)  was  narrowed  to  0.9  point  on  the  post-test 
(43.0  versus  42.1). 

Naval  Science  1,  Sanger  High  School 

At  Sanger,  the  0.3  point  difference  in  the  NS  1  groups’  scores  on  the 
pre-test  is  not  statistically  significant  (p  =  0.8672),  but  the  2.8  point 
difference  on  the  post-test  in  favor  of  the  NMIS  group  is  statistically 
significant  below  the  10  percent  level  (p  =  0.0679).  In  other  words, 
the  groups  had  a  comparable  understanding  of  the  course  material  at 
the  beginning  of  the  study;  however,  there  is  a  less  than  10  percent 
probability  that  the  NMIS  group’s  higher  average  at  the  end  of  the 
study  is  a  random  event. 
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American  Government,  Sanger  High  School 

In  the  American  Government  course,  the  NMIS  students  had,  on  av¬ 
erage,  a  lower  pre-test  score  than  students  in  the  Traditional  group, 
and  the  difference  is  statistically  significant  below  the  10  percent 
level  (p  =  0.0796).  The  NMIS  group’s  average  score  increased  29.7 
percent  (5.1  points)  from  pre-test  to  post-test  whereas  the  Tradi¬ 
tional  group’s  average  score  decreased  7.3  percent  (-1.4  points).  The 
difference  in  the  groups’  post-test  scores  is  statistically  significant 
(p  =  0.0015).  The  increase  in  the  NMIS  students’  average  score  is 
consistent  with  the  gains  achieved  by  NMIS  groups  taking  NS  1,  but 
the  decrease  in  the  Traditional  group’s  score  is  surprising.1  We 
return  to  these  outcomes  in  the  next  chapter  of  this  report. 

NMIS  Effect  Relative  to  Ethnicity 

At  Sanger,  analysis  of  the  change  in  NS  1  test  scores  by  ethnicity 
highlights  a  potentially  important  contribution  of  the  NMIS.  His¬ 
panic  students  in  both  groups  had  significantly  lower  pre-test  scores 
than  did  students  of  other  ethnicities.  However,  Hispanic  students 
in  the  NMIS  group  achieved  a  23.4  percent  gain  from  pre-test  to 
post-test  (slightly  ahead  of  the  22.1  percent  gain  for  NMIS  students  of 
all  other  ethnic  groups).  In  contrast,  Hispanic  students  in  the 
Traditional  group  achieved  a  gain  of  15.5  percent  (comparable  to 
students  of  other  ethnicities  who  were  in  the  Traditional  group). 

It  is  not  meaningful  to  analyze  the  scores  at  Lemoore  by  ethnicity  be¬ 
cause  there  are  insufficient  numbers  of  students  in  each  category  to 
provide  statistical  validity. 

STUDENTS’  ATTITUDES 

Overall,  the  124  students  who  participated  in  the  focus  group  ses¬ 
sions  at  Lemoore  and  Sanger  were  strongly  positive  and  overwhelm¬ 
ingly  enthusiastic  about  the  NMIS.  There  was  a  high  level  of  consis¬ 
tency  among  their  comments.  In  sum,  they  reported  that  they  prefer 


.The  decline  of  the  Traditional  group’s  average  score  from  the  pre-test  to  the  post-test 
is  an  anomaly.  We  suggest  that  readers  should  exercise  care  when  using  data  per¬ 
taining  to  the  American  Government  course. 
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NJROTC  courses  using  the  system  to  courses  using  traditional  meth¬ 
ods  of  instruction.  They  said  the  NMIS  makes  their  course  work 
“more  understandable,”  “easier,”  “enjoyable,”  and  “fun."  In  each 
group  that  had  used  the  system,  the  overwhelming  majority  of  the 
students  said  they  would  recommend  classes  using  it  to  their  friends. 

Four  central  themes  stand  out  in  the  students’  comments.  The  first 
theme  relates  to  the  structure  that  is  imposed  by  the  system  on  the 
instructor's  lecture  and  the  students’  work.  The  second  major  theme 
relates  to  attractiveness  of  the  NMIS  technology.  The  third  theme 
concerns  the  ease  with  which  the  system  is  learned  and  used.  The 
fourth  theme  concerns  the  manner  in  which  students’  perceptions  of 
the  system  change  as  they  gain  experience  with  it. 

The  quotes  below  are  representative  of  comments  made  by  the  ma¬ 
jority  of  the  students.  To  respect  the  privacy  of  the  students  and 
their  instructors,  comments  are  not  identified  by  school  or  by  course. 

The  System  Organizes  Work  Efficiently 

Students  said  the  NMIS  organizes  their  instructors,  the  class  periods, 
and  their  own  work.  They  described  the  structuring  that  is  imposed 
by  the  NMIS  as  a  positive  attribute. 

They  said  they  like  receiving  the  class  schedule  at  the  start  of  the 
week.  Most  said  they  copy  it  to  their  notes,  so  they  can  refer  to  it  as 
the  week  progresses. 

Relative  to  classes  taught  with  traditional  methods,  students  said  the 
lectures  in  NMIS  classes  are  easier  to  follow  and  understand  because 
the  system  organizes  instructors’  presentations  in  a  manner  that 
highlights  the  main  issues.2 

In  other  classes  you  don’t  know  what  the  instructor  is  saying  that  s 
important. 

Students  reported  that  there  is  more  time  in  NMIS  classes  for  dis¬ 
cussion — which  they  said  is  particularly  helpful — and  they  attributed 


2 “Last  year  [in  a  NJROTC  course  not  using  the  NMIS]  sometimes  the  pictures  on  the 
laser  disc  didn’t  make  sense.  Now,  we  know  what  the  pictures  mean. 
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this  advantage  to  the  system.  They  said  the  NMIS  requires  instruc¬ 
tors  to  come  to  class  with  their  presentation  already  prepared,  and  it 
enables  instructors  to  deliver  the  lesson  much  faster  than  traditional 
methods  (e.g.,  time  is  not  taken  to  write  information  on  a  blackboard, 
to  turn  pages  in  a  book,  to  recall  the  next  point  to  be  addressed). 

Students  said  that  the  outline  shell  provided  by  the  NMIS  serves  to 
focus  the  notes  they  write  on  lectures  and  course  materials.  This  fo¬ 
cus,  they  said,  helps  them  when  they  review  for  tests  and  when  they 
write  essays  from  their  notes.  However,  one  senior  commented: 

I  don't  know  if  it  really  prepares  you  for  college.  Maybe  there’s  too 
much  guidance. 

but  another  student  replied: 

Here  [in  an  NMIS  class],  I  think  I  leam  the  subject  better,  and  I  learn 
better  how  to  outline  and  [identify  key  points]. 

We  asked  students  how  many  notes  they  take,  and  the  responses 
ranged  from  “a  lot”  to  “barely  any.”  Our  observations  of  the  NMIS 
classes  and  our  review  of  the  students’  note  files  indicated  that  stu¬ 
dents  using  the  system,  on  average,  take  at  least  a  typewritten  page  of 
notes  during  each  lecture  and  on  each  reading  assignment.  The 
amount  of  notes  varies  by  course  and  instructor,  as  well  as  by  stu¬ 
dent.  We  observed  far  fewer  notes  being  taken,  on  average,  in  classes 
using  traditional  methods  of  instruction.3 

Students  also  commented  that  the  NMIS  organizes  their  work  physi¬ 
cally.  With  the  NMIS,  the  material  is  "right  there,”  readily  available 
for  review.4  In  addition,  students  said  they  like  the  system  because  it 
provides  all  the  materials  they  need;  in  contrast,  in  other  classes, 
students  often  forget  to  bring  textbooks,  paper,  pens,  homework,  etc. 

You  don’t  need  a  lot  of  supplies  for  this  class.  You  just  need  fingers. 


3 A  student  commented,  “In  some  [non-NMIS]  classes,  some  kids  just  listen  and  learn 
without  taking  notes.  It’s  not  the  same  here.’’ 

4“You  see  kids  [in  school]  with  binders  with  things  falling  all  over  the  place.  Here 
things  are  organized.  I  can  find  things  when  I  need  them.” 
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The  Technology  Is  Compelling 

The  second  major  theme  of  the  focus  groups  relates  to  the  attrac¬ 
tiveness  of  the  NMIS  technology. 

Studying  gets  old  quick.  With  the  computer,  you  have  more  ways  to 
learn. 

More  specifically,  students  commented  about  the  appeal  of  the 
system  in  terms  of  the  course  readings,  lectures,  students’  notes,  and 
exams. 

It’s  easier  to  memorize  off  a  TV  than  a  piece  of  paper — it’s  just  eas¬ 
ier.  You  can  look  at  a  piece  of  paper  over  and  over  and  not  remem¬ 
ber  it.  It’s  like  a  video  game— you  remember. 

Students  reported  they  find  course  material  more  engaging  when  it  is 
presented  via  the  system  rather  than  via  a  textbook.  They  perceive 
that,  relative  to  books,  (1)  it  is  easier  to  scan  the  material,  (2)  it  is 
easier  to  locate  information,  particularly  when  reviewing  for  a  test, 
and  (3)  hypertext  links  provide  faster  access  to  definitions.5  Stu¬ 
dents’  descriptions  of  the  hypertext  links  suggest  the  interplay  of  the 
technology  with  the  mechanics  of  reading. 

A  picture  makes  you  want  to  read  the  paragraph — want  to  see 
what’s  behind  the  picture. 

Sure,  I  look  up  the  definitions  on  the  system  if  I  don’t  know  them. 

It’s  easy.  But,  I  don’t  look  up  words  I  don’t  know  in  textbooks.  That 
takes  too  much  time.  It’s  too  hard. 

Students  also  said  they  find  the  questions  at  the  end  of  the  units  in 
the  NS  1  text  are  helpful,  and  they  like  being  able  to  copy  the  ques¬ 
tions  into  their  notes  with  just  a  few  keystrokes. 

Students  reported  that  instructors’  lectures  are  more  interesting  and 
more  enjoyable  in  NMIS  classes  than  lectures  in  traditional  courses 


5Most  students  reported  that  they  take  advantage  of  the  hypertext  links  to  pictures  and 
other  graphics,  but  they  use  the  links  to  the  glossary  less  frequently  because  they 
understand  some  of  the  words,  and  do  not  need  to  refer  to  the  definition. 
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because  NMIS  lectures  use  graphical  materials  more  effectively. 
More  specifically,  lectures  in  NMIS  classes  integrate  a  variety  of  ma¬ 
terial,  e.g.,  pictures,  other  graphics,  and  articles  from  the  morning’s 
newspaper. 

You  are  getting  the  same  things  in  different  ways.  It  makes  it  easier 
to  understand  and  remember  them. 

However,  one  student  commented: 

There  are  lots  of  still  pictures.  Some  are  dull.  If  he’s  talking  about  a 
captain,  he  just  shows  a  picture  of  a  captain  just  standing  there. 

The  Instructor’s  Notes  featured  prominently  in  the  students’  com¬ 
ments. 

It’s  easier  than  the  instructor  writing  on  a  chalk  board — you  can 
scroll  back  if  you  want  [if  you  miss  something] — and  you  can  copy  it 
[with  a  couple  of  keystrokes]. 

I’d  miss  a  point  [in  other  classes,  and]  I’d  let  it  go.  Here  I  have  his 
notes. 

Students’  comments  indicate  that  the  NMIS  media  offer  new  oppor¬ 
tunities  to  present  material.  For  example,  they  cited  software  that 
displays  a  rotating  Earth,  locks  on  an  area,  and  zeros  into  a  locality 
(see  the  subsection  "Listening  to  the  Lecture  and  Taking  Notes,” 
Chapter  Two). 

We  asked  students  whether  traditional  visual  aids  (e.g.,  transparen¬ 
cies,  films,  etc.)  are  used  effectively  in  non-NMIS  courses.  Students 
told  us  that,  typically,  visual  aids  are  not  well-integrated  into  lectures 
in  non-NMIS  classes,  if,  indeed,  they  are  used  at  all. 

[Other  classes]  are  really  boring.  The  instructors  just  stand  up  there 
lecturing. 

As  noted  above,  students  commented  favorably  that  the  system  pro¬ 
vides  more  class  time  for  discussion.  They  also  said  the  system  is 
used  actively  to  facilitate  discussion  and  review.  On  one  occasion, 
we  observed  an  instructor  pause  during  his  lecture  to  ask  questions 
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about  related  material,  which  the  class  had  covered  previously.  One 
student  was  unable  to  reply  to  a  question.  The  instructor  said  that  all 
of  the  students  should  know  the  answer  to  this  question,  and  he  di¬ 
rected  the  student  who  could  not  respond  to  call  up  his  notes  on  the 
computer  and  find  the  answer.  Most  of  the  other  students,  perhaps 
anticipating  that  the  question  might  be  directed  to  them,  reviewed 
their  own  notes  while  their  classmate  was  thus  engaged. 

Students  also  said  they  enjoy  using  the  “Jeopardy”  game  as  a  review 
device  (see  the  subsection  “Classroom  Discussion,”  Chapter  Two). 
During  our  visits  to  the  classes,  we  observed  the  class  applauding 
and  whistling  if  a  student’s  answer  was  correct.  Otherwise,  a  barrage 
of  good-natured  catcalls  and  banter  followed.  The  game  led  readily 
to  discussion  when  it  was  appropriate  to  reinforce,  amplify,  or  cor¬ 
rect  a  student’s  answer. 

By  far  the  majority  of  the  students  in  the  focus  groups  reported  that 
they  prefer  using  the  NMIS  to  take  notes  on  the  reading  materials 
and  lectures  rather  than  writing  notes  by  hand.  They  said  typing  is 
faster  and  easier  than  writing  by  hand  and,  many  students  stressed, 
they  often  are  not  able  to  read  their  handwriting  when  they  review 
their  notes.  They  explained  that  students  are  under  pressure  to  write 
rapidly  during  non-NMIS  lectures,  and  handwritten  notes  often  are 
difficult  to  read  because  they  are  terse  and  messy,  and  the  penman¬ 
ship  is  careless. 

In  addition,  students  said,  with  the  NMIS  they  can  easily  use  colors 
to  highlight  key  concepts. 

It’s  like  memory  cues.  You  can’t  find  them  in  a  regular  classroom 
When  I  reviewed,  I  saw  what  was  important  my  cues  helped.  I 

can’t  do  that  with  hard  copy  notes.  You  have  to  have  a  lot  of  pens. 

Besides,  I’m  too  sloppy. 

Students  reported  that  they  prefer  to  take  tests  on  the  computer 
rather  than  use  paper  and  pencil,  in  the  main,  because  the  system 
provides  immediate  feedback.6  Students  noted. 


6 An  instmctor  commented,  “There’s  instant  gratification  for  the  students.  Students 
like  to  know  how  they  did.  I’ve  found  if  you  give  them  the  results  of  a  test  a  week  later, 
they’ve  lost  interest.” 
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In  other  courses,  you  have  to  wait  a  week  to  find  out  how  you  did. 
And  by  then  you’ve  forgotten  what  the  test  was  all  about. 


Students  Find  the  NMIS  Easy  to  Learn  and  Use 

Students  in  the  NMIS  groups  reported  that  they  were  able  to  use  the 
system  without  difficulty,  and  the  one-week  orientation  to  the  NMIS 
was  sufficient  time  to  learn  how  to  use  it.  To  use  the  system,  stu¬ 
dents  must  be  able  to  perform  several  basic  tasks,  which  are 
summarized  in  Appendix  E;  these  tasks  were  identified  by  Captain 
Nicholson,  the  developer  of  the  system. 

Most  of  the  students  reported  that  they  had  taken  or  were  taking  the 
school’s  keyboarding  course  and  were  comfortable  typing.  Those 
who  were  not  touch-typists  said  that  the  system  was  “more  fun”  once 
they  learned  to  type,  but  that  “People  go  at  their  own  pace.  There’s 
no  real  problem  if  they  can’t  type  too  well.”7 

During  the  focus  group  sessions,  we  noted  that  most  of  the  students 
reported  that  they  had  considerable  prior  experience  with  comput¬ 
ers.  In  the  main,  the  students  were  introduced  to  computers  at 
school,  but  many  of  them  also  gained  experience  with  computer 
systems  at  home.  Only  a  few  students  reported  that  the  NMIS  or  the 
school’s  keyboarding  course  was  their  first  exposure  to  computers. 
Lemoore  students  said  they  began  using  computers  at  an  average  age 
of  approximately  10.5  years.  At  Sanger,  the  average  age  was  approx¬ 
imately  11  years.  Some  of  the  students  reported  they  had  begun  us¬ 
ing  computers  in  the  early  years  of  grade  school,  and  at  least  two  of 
them  had  first  used  a  computer  while  they  were  in  kindergarten. 
Playing  games  constituted  much  of  the  students’  early  experience 
with  computers.  However,  some  students  reported  using  computers 
for  math  exercises,  to  write  essays,  etc. 


instructors  reported  that  most  students  were  able  to  use  the  system  without  diffi¬ 
culty.  “I  haven’t  seen  any  student  have  problems  with  the  system.  When  we  start, 
about  one-third  of  the  class  can’t  keep  up,  but  that  third  improves,  and  they  are  able 
to  keep  up  within  about  six  weeks.” 
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Students’  Perceptions  of  the  System  Evolve  as  They  Gain 
Experience  with  It 

We  talked  with  students  who  had  never  used  the  NMIS  in  class,  but 
who  were  somewhat  familiar  with  the  system  through  discussions 
with  other  students.  In  addition,  some  of  them  had  used  the  NMIS 
computers  to  type  assignments  for  other  courses.  Their  comments 
expressed  polar  views  of  information  technology,  e.g.,  some  lauded 
the  opportunity  to  learn  new  skills  that  they  perceived  as  being  re¬ 
quired  in  the  workplace  and  in  society,  but  others  were  wary  of  sys¬ 
tem-imposed  isolation  (i.e.,  working  with  a  machine  instead  of  peo¬ 
ple),  and  were  concerned  about  their  technological  self-efficacy.  It 
appears  these  students  were  evaluating  the  NMIS  based  on  their  ex¬ 
perience  with  other  computer  systems  and/ or  with  their  projections 
of  what  working  with  the  NMIS  entails. 

I  hate  my  computer  class  [the  school’s  keyboarding  course].  You  sit 
behind  a  computer  for  a  long  time  [the  school  has  95  minute  peri¬ 
ods].  Two  back-to-back  computer  classes  would  be  hard  to  take.8 


When  you  write  [by  hand] ,  you  have  time  to  think. 

We  also  talked  with  a  group  of  students  who  had  started  using  the 
NMIS  late  in  the  school  term.  We  call  them  the  “new  users.”  We 
conducted  two  focus  group  sessions  with  these  students.  The  first 
session  was  held  approximately  one  month  after  they  started  using 
the  system,  and  the  second  was  convened  three  months  later. 

During  the  first  session,  we  were  particularly  interested  to  hear  new 
users’  perceptions  of  the  system  contemporaneous  with  their  initial 
exposure  to  it.  Over  half  of  them  said  they  liked  using  the  system, 
and  their  comments  were  similar  to  the  comments  made  by  students 
who  had  used  the  NMIS  for  at  least  one  school  term.  Most  of  the  new 
users  said  they  would  recommend  a  class  using  the  system  to  their 
friends;  however,  one  student  commented: 


8Typically,  when  we  asked  groups  of  experienced  NMIS  users  if  they  would  like  a  day 
of  class  when  they  did  not  use  the  computer,  the  students  answered  loudly,  “NO!” 
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I  wouldn’t  if  they  don’t  have  good  attention  skills,  if  they  hate  com¬ 
puters,  don’t  have  experience  with  them,  and  don’t  like  to  try  new 
things. 

The  group  reported  that  it  had  used  the  system  to  take  two  tests.  Ap¬ 
proximately  one-third  of  the  students  liked  using  the  computer,  one- 
third  preferred  paper  and  pencil  tests,  and  one-third  said  they  did 
not  care  which  medium  was  used.  All  of  the  students,  except  one, 
said  they  liked  getting  the  grade  right  away.  That  one  student  said: 

I  get  frustrated  when  I  get  one  wrong.  I’d  rather  not  know. 

Approximately  half  of  the  new  users  made  neutral  or  negative 
comments  comparable  to  those  we  had  heard  from  the  non-users. 
Three  of  the  students  had  a  very  strong  preference  for  textbooks  and 
for  writing  notes  by  hand,  in  part,  they  said,  because  they  lacked 
keyboard  skills  and  in  part  because  they  preferred  to  use  hard  copy 
materials  outside  the  normal  class  hours. 

Typing  doesn’t  go  as  fast  for  me  as  handwriting.  When  he  lectures 
us,  I  don’t  hear  anything  he’s  saying.  I  read  in  the  book  later  be¬ 
cause  I’m  too  busy  typing. 

We  found  that  some  of  the  new  users  were  not  yet  familiar  with  all  of 
the  system  options  available  to  them  (e.g.,  finding  out  how  much 
time  was  left  to  complete  a  test,  or  how  to  use  the  hypertext  links). 

During  the  second  focus  group  session  with  the  new  users,  we  ex¬ 
plored  whether  and  how  their  impressions  changed  as  they  gained 
experience  with  the  system.  We  found  continued  support  for  the 
system  among  the  students  who  favored  it  initially.  We  found  there 
was  more  support  for  the  system  among  the  students  who  had  been 
neutral  or  negative.  Approximately  two-thirds  of  the  students  re¬ 
ported  they  felt  better  about  using  the  system,  one-third  said  they  felt 
about  the  same,  and  no  one  reported  feeling  worse  about  using  it. 

Only  one  student  continued  to  strongly  prefer  textbooks  and  tradi¬ 
tional  methods  of  instruction  to  the  system. 
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Students’  Recommendations 

In  general,  students  reported  that  they  like  the  system  as  it  is. 

There’s  nothing  we  really  dislike. 

However,  their  comments  identified  three  aspects  that  merit  atten¬ 
tion:  hard  copy  output,  file  protection,  and  automatic  file-saves. 

Some  students  said  they  printed  hard  copy  of  their  work  on  occasion. 
However,  most  of  the  students  reported  that  they  cannot  print  files, 
but  would  like  to  be  able  to  output  their  notes,  etc.,  in  hardcopy,  and 
read  them  outside  of  the  class.  A  few  students  commented  they 
would  like  to  copy  their  files  to  a  disk,  and  then  read  their  notes,  etc., 
on  their  computer  at  home.  The  policy  regarding  printing  appears  to 
be  set  by  the  instructors.  One  instructor  commented,  “There’s  a 
problem  letting  the  students  make  hardcopy  notes,  mainly,  the  big 
demand  for  paper.”  The  instructor  also  was  concerned  about  system 
security.  He  noted  that,  “We  could  transfer  files  to  students’  floppy 
disk,  but  I  worry  that  if  they  brought  in  a  disk,  they  might  download 
something  from  it  that  would  have  a  virus.” 

Several  students  reported  they  had  lost  files  because  someone 
deleted  them.  This  may  have  been  accidental.  However,  in  some 
courses,  students  were  entering  other  students’  files  without  per¬ 
mission,  and  leaving  notes. 

About  one-third  of  the  students  in  one  group  had  lost  files  due  to 
server-related  problems.  The  problems  appear  to  have  been  re¬ 
solved,  but,  understandably,  the  crashes  were  well-remembered. 

Other  comments  made  during  the  student  focus  groups  suggested 
enhancements  such  as  including  more  pictures,  particularly  color 
pictures,  adding  more  movies,  playing  “Jeopardy”  more  often, 
learning  more  about  the  system  (e.g.,  “shortcuts  to  use  as  we  gain 
experience”),  getting  a  more  up-to-date  text,  and  “hook  it  up  to  the 
Internet.” 


INSTRUCTORS’  WORKLOAD 

Experienced  instructors  spoke,  in  a  sense,  with  one  voice.  Relative  to 
traditional  methods  of  instruction,  they  find  with  the  NMIS  that 
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lessons  are  easier  to  prepare  and  modify,  tests  are  scored  more 
quickly,  and  their  interaction  with  students  is  more  satisfying.  The 
comments  below  are  representative  of  their  responses. 

Class  Preparation  Takes  Less  Time  with  the  System 

Once  the  initial  course  material  has  been  input  into  the  system,  in¬ 
structors  find  that  their  lectures  can  be  modified,  updated,  and  sup¬ 
plemented  more  easily  than  with  traditional  methods  of  instruction. 

I  take  the  basic  lesson  guide  and  tailor  it  to  my  particular  needs.  I 
add  buttons  to  my  Notes  where  I  need  them  for  the  laser  and  VHS. 
Once  it’s  all  there,  it’s  all  there.  I  use  each  part  of  the  Instructor’s 
Notes  just  once  during  a  year— but  I  used  it  last  year,  I  used  it  this 
year,  and  I’ll  use  it  next  year.  I  may  update  it,  and  if  there’s  a  mis¬ 
take,  I’ll  correct  it.  But  that  doesn’t  take  much  effort.  It  is  much 
easier  to  modify  than  written  materials.  It’s  simple  to  cut  and  paste, 
reorganize,  add  a  picture,  etc.  There  are  so  many  advantages  like 
that  to  using  a  computer.  The  traditional  methods  of  instruction 
are  very  time-consuming. 


Reviewing  Students’  Work  Is  Easier 

Instructors  reported  major  savings  of  time  by  using  the  NMIS  to  pre¬ 
pare,  administer,  and  grade  tests.  An  added  benefit  of  using  the  sys¬ 
tem  is  being  able  to  determine  easily  and  immediately  “who’s  doing 
what  in  class,  who  needs  help.” 

Instructors  also  commented  that  students’  work  is  neat  when  it  is 
done  using  the  system.  Neatness  is  not  a  trivial  issue,  and  it  affects 
all  types  of  the  students’  work  (e.g.,  homework  and  essays).  Instruc¬ 
tors  reported  that  it  can  take  several  weeks  to  become  accustomed  to 
some  students’  handwriting.  Even  then,  handwritten  materials  often 
are  difficult  to  read.9  With  the  NMIS, 

Their  typing  is  clear.  All  this  saves  time.  I  have  more  time  at  home. 

I  have  more  time  to  assign  and  correct  essays — also,  “Mark  Up”  [the 


^Similarly,  students  commented  that  they  like  to  have  their  work  annotated  via  the 
computer  because  it  is  difficult  for  them  to  read  some  instructors'  handwriting. 
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software  program  used  by  instructors  to  annotate  the  students 
work]  is  much  faster  than  pen  and  pencil. 

As  the  preceding  quote  indicates,  some  instructors  prefer  to  grade 
students’  work  at  home  where,  they  report,  there  are  fewer  distrac¬ 
tions  and  they  can  work  more  productively  and  more  comfortably 
than  in  the  school  setting.  With  the  NMIS,  they  are  able  to  load  stu¬ 
dents’  work  onto  their  laptop  computer  or  a  diskette,  use  their  com¬ 
puter  at  home  to  annotate  the  students’  work,  then,  back  at  school, 
download  the  corrected  material  to  the  students’  computers.  This 
process  saves  time  outside  the  classroom  as  well  as  in  it,  as  discussed 
below. 

The  NMIS  Facilitates  Classroom  Management 

Instructors  reported  that  the  NMIS  simplifies  many  routine  class 
management  tasks.  For  example,  the  system  enables  instructors  to 
collect  and  return  students’  work  literally  instantaneously  without 
the  loss  of  student  attention.10 

Instructors  said  the  system  organizes  the  students,  and  enables  in¬ 
structors  to  pace  the  class  more  effectively  than  do  traditional  meth¬ 
ods. 

As  soon  as  you  say,  “Let’s  start,”  you  get  going.  No  time  is  taken  to 
get  pencils  and  books.  There’s  no  time  spent  with  excuses  like,  “I 
didn’t  bring  my  homework.”  You  can  start  class  at  the  click  of  a 
button.  You  can  control  the  pace  of  the  class  very  easily. 

The  system  provides  efficient  media  to  deliver  instruction,  which 
saves  time  during  class.  We  asked  instructors  if  it  would  be  equally 
time-saving  to,  for  example,  write/ draw  on  transparencies  before 
class.  They  responded  that  it  is  easier  to  create  materials  in  the  NMIS 
than  on  transparencies;  moreover,  it  can  be  a  logistical  challenge  to 
integrate  traditional  media,  e.g.,  to  go  from  the  blackboard  to  the 
transparency  projector  to  the  audio-visual  device  particularly  if 
they  want  to  combine  a  series  of  brief  segments  such  as  a  key  word 


10“An  instructor  noted,  “Kids  lose  their  attention  when  you  have  to  take  a  break  to 
collect  homework  or  hand  out  graded  papers  or  set  up  equipment.” 
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written  on  the  blackboard,  followed  by  one  picture  displayed  from 
the  laser  disc  player,  followed  by  text  projected  from  a  transparency. 

The  NMIS,  instructors  reported,  enables  them  to  manage  the  class 
more  effectively.  First,  the  system  is  less  distracting  to  use  than  tra¬ 
ditional  instructional  media,  so  instructors  can  be  more  responsive 
to  the  class  and  maintain  more  eye  contact  with  the  students.  Sec¬ 
ond,  it  enables  instructors,  particularly  when  they  are  using  a  cord¬ 
less  mouse  (see  the  subsection  “Listening  to  the  Lecture  and  Taking 
Notes,”  Chapter  Two),  to  walk  around  the  classroom  and  observe 
and  assist  individual  students  while  they  are  working.11  Third,  in¬ 
structors  reported  that  the  system  keeps  students  organized  and  on- 
track  which  saves  class  time  and  makes  the  class  easier  and  more 
satisfying  to  teach. 

When  they’re  working  with  computers,  they  aren’t  bored  as  they  are 

in  the  other  classes.  It’s  a  lot  easier  to  keep  them  going. 


With  the  system,  the  students  must  take  notes  all  the  time.  Manu¬ 
ally,  they  stop. 

The  system  forces  the  kids  to  do  the  work.  They  can’t  come  in  with 
answers  they  copied  from  others  just  before  the  class.  That  hap¬ 
pened  before!  It  forces  them  to  do  it  themselves. 

Instructors  also  reported  that  the  system  is  more  effective  than  tradi¬ 
tional  methods  when  they  are  counseling  students  and  parents.  A 
student’s  work  is  easy  to  locate,  display,  and  compare  with  other  stu¬ 
dents’  work — and,  if  needed,  a  copy  can  be  mailed  to  the  parents  for 
their  review. 

The  System  Helps  Instructors  Enhance  the  Curriculum 

Instructors’  comments  suggest  that,  at  least  in  part,  the  time  saved  by 
the  system  is  reinvested  in  the  curriculum.  Instructors  report  they 
add  content  to  their  course  because  the  lectures  move  more  quickly 
than  when  traditional  methods  are  used.  They  download  supple- 


1  ^‘Before,  I  was  stuck  at  the  front  of  the  room  fiddling  with  paperwork  or  playing  with 
the  VCR  or  the  OH  projector.” 
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mental  materials  from  the  Internet,  and  they  can  easily  “include 
material  from  magazines  and  the  morning’s  newspaper.” 

It  takes  me  seven  minutes  to  scan,  add  comments,  and  load  it  onto 
the  students’  computers.  It  is  hard  for  teachers  in  other  [non- 
NMIS]  courses  to  do  all  that  is  required — you  know,  cut  and  paste 
an  article  to  8.5  by  11  inch  paper,  type  or  print  notes  in  the  margins, 
take  the  sheet(s)  to  the  school’s  copier  [in  the  main  building],  wait 
until  the  copier  is  available,  make  thirty  copies,  then  return  to  the 
classroom,  and  then  distribute  the  material  in  class. 


Homework  and  Special  Projects 

Homework  assignments  are  minimal  and,  typically,  utilize  texts  or 
other  material  independent  of  the  NMIS;  “homework  is  almost  al¬ 
ways  a  matter  of  reading  something  at  home,  coming  in  and  writing 
notes  about  it  on  the  system  during  class.” 

Instructors  also  have  students  write  essays  using  their  notes  on  the 
course  materials  and/or  the  instructor’s  lectures.  Instructors  com¬ 
mented  that  students  quickly  learn  the  benefit  of  taking  good  notes. 

Some  instructors  assign  special  projects  (either  group  exercises  or 
individual  exercises),  and  they  may  load  resource  data  for  these  as¬ 
signments  into  NMIS.  For  example,  one  instructor  asked  students  to 
write  an  article  about  President  Lincoln’s  assassination  as  if  they 
were  a  reporter  for  an  English  newspaper.  The  article  had  to  be  in 
newspaper  format,  and  it  had  to  include  a  graphic.  For  this  assign¬ 
ment,  the  instructor  loaded  information  and  graphics  into  NMIS. 
Some  students  added  supplemental  materials  from  other  sources 
(which  they  entered  into  NMIS  using  the  NJROTC  scanner).  In  an¬ 
other  instance,  an  instructor  loaded  recent  articles  about  the  pro¬ 
posed  flat  tax,  asked  students  to  read  the  material,  discuss  it,  and 
then  write  letters  expressing  their  views  to  the  U.S.  Senators  from 
California. 

Learning  to  Use  the  NMIS 

Instructors  divide  the  process  of  learning  to  use  the  system  into  two 
phases.  The  first  phase  consists  of  learning  how  to  deliver  course 
materials  that  are  loaded  into  the  system,  generate  some  additional 
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material,  and  administer  tests  via  the  system.  They  estimated  that 
this  set  of  skills  takes  approximately  two  weeks  to  acquire. 

In  one  week,  you  could  teach  a  relative  newbie  to  be  self-sufficient 
in  terms  of  presenting  and  doing  some  creation  of  class  materials. 

In  two  weeks,  a  new  instructor  would  be  comfortable  presenting  in¬ 
formation  given  to  him,  using  buttons,  coordinating  the  laser  disk 
with  the  VCR. 

The  second  phase  consists  of  learning  to  develop  a  course  from 
scratch,  i.e.,  to  load  text,  create  Instructor’s  Notes  and  exams,  and 
similar  tasks.  Typically,  that  phase  was  estimated  to  take  an 
additional  two  weeks. 

However,  a  new  instructor  said  he  was  just  becoming  effective  with 
the  system  after  approximately  12  weeks  of  teaching.  He  noted  that 
he  had  to  become  familiar  with  computers,  the  course  material,  and 
the  multi-media  approach. 

I  couldn’t  have  [used  HyperCard]  to  set  up  the  Instructor’s  Notes 
and  I  couldn’t  have  set  the  buttons  when  I  started.  There  was  no 
time  to  learn  in  the  few  weeks  from  when  I  got  here  until  I  started 
teaching.  I  had  to  come  up  to  speed  with  computers — and  with  the 
course  materials.  Anyone  can  teach  class  [after  the  NJROTC  orien¬ 
tation  program].  Here  I  had  to  learn  to  use  multimedia  to  do  it.  I 
learned  with  OJT  [on-job  training].  I’m  only  now  becoming  effec¬ 
tive. 


Would  Instructors  Recommend  the  NMIS  to  Other  NJROTC 
Units? 

Instructors  stated  unanimously  that  they  would  recommend  in¬ 
stalling  the  NMIS  at  other  NJROTC  units.  They  cited  advantages  it 
provides  to  students  and  instructors,  mainly  in  terms  of  the  delivery 
of  instruction,  motivation  and  learning,  familiarizing  students  with 
information  systems  technology,  the  potential  to  innovate  in  the 
classroom,  and  the  ability  to  attract  more  and  better  students  to  the 
NJROTC  program. 

Instructors  commented  that  technical  support  should  be  provided  to 
new  installations — both  initially  when  instructors  are  learning  to  use 
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it  and  subsequently  as  they  explore  the  system’s  advanced  features 
and  embed  innovations  in  their  course  curriculum.  This  latter  type 
of  support  appears  to  be  substantive  as  well  as  technical. 

About  50%  of  our  questions  .  . .  concern  extensions  to  the  system, 
"Can  we  do  this . . .  ?”  The  other  50%  concern  . .  .the  use  of  technol¬ 
ogy  in  education,  that  is,  sharing  professional  opinions  about  the 
use  of  the  system . . .  and  integrating  new  software  into  the  system. 


EFFECT  OF  THE  NMIS  ON  SCHOOL  CULTURE 

The  Assistant  Superintendent  of  the  Lemoore  School  District  was 
asked,  “Does  the  NJROTC  system  [the  NMIS]  add  something  that  is 
not  present  at  Lemoore  High  School?”  His  reply  was  an  emphatic, 
“Yes!”  He  noted  that  even  though  the  school  has  integrated  infor¬ 
mation  systems  technology  into  its  courses  to  a  relatively  advanced 
degree,  the  NJROTC  unit  is  an  exemplar.  It  is  perceived  as  being  on 
the  forefront  of  change  because  of  its  ability  to  apply  an  important 
technology.  Similarly,  students  at  both  schools  commented  that 
other  "computer  labs”  in  their  school  have  generic  software  (such  as 
word  processors),  but  the  NJROTC  system  is  distinctive  and  benefi¬ 
cial  because  it  addresses  specific  course  curriculum. 

James  Paige,  an  Apple  Computer,  Inc.,  district  manager,  wrote12  the 
following  about  Lemoore  (his  comments  apply  equally  to  Sanger): 


As  you  might  imagine,  in  Northern  California  we  see  virtually  hun¬ 
dreds  of  schools  each  year  set  up  similar  labs  that  target  all  facets  of 
elementary  and  secondary  education.  The  lab  at  Lemoore  is  suc¬ 
cessful  from  our  viewpoint  for  the  following  reasons: 


1.  Often,  schools  . . .  utilize  the  computer  as  an  (expensive)  electric 
typewriter  or  as  a  "drill”  device.  [At  Lemoore,  students  are  taught 
to]  utilize  the  computer  as  a  tool  to  address  many  aspects  of  learn¬ 
ing  and  communicating.  Today’s  students  are  motivated  by  this 
approach  and  as  a  result  put  forth  extra  efforts  in  their  work. 


12Personal  correspondence  dated  November  6, 1995. 
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2.  [The  system  is  integrated  into  the  curriculum.]  Rather  than  make 
the  computer  the  center  of  focus,  the  areas  of  learning  continue  to 
be  the  major  focal  point  with  the  Macintosh  computers  used  to 
augment  the  learning  process. 

The  NJROTC  units  at  Lemoore  and  Sanger  have  operated,  as  it  were, 
a  type  of  outreach  program  to  other  departments  of  the  school.  For 
example,  NJROTC  personnel  at  Lemoore  gave  school  personnel  a 
demonstration  of  the  testing  software  incorporated  into  the  NMIS. 
As  a  result,  the  school  obtained  a  site  license  for  the  software,  and  the 
faculty  are  looking  to  NJROTC  personnel  for  guidance  in  using  it. 
Sanger  NJROTC  incorporated  the  “Personal  Trainer  for  the  SAT”  into 
its  program,  and  has  invited  classes  from  the  English  and  Mathemat¬ 
ics  departments  to  use  it.13 

The  maintenance  of  the  equipment  is  a  hallmark  of  the  NMIS  instal¬ 
lations  at  both  schools,  and  it  is  apparent  to  students14  and  visitors 
alike.15  A  student  commented, 

When  you  come  in  here,  it’s  like  nothing  you’ve  seen  on  campus 
before.  It’s,  “oh  my  gosh!” 


13An  NJROTC  instructor  commented,  “I  want  them  to  have  options.  I  want  them  to 
have  lots  of  choices.  That's  why  I  have  the  SAT  preparation  software.  I  couldn’t  have 
done  this  without  the  NMIS  equipment.  You  can’t  get  them  interested  in  a  SAT  prep 
guide  from  a  bookstore.  They  wouldn’t  do  the  prep  without  a  computer.  They  would 
take  it  cold." 

14A  student  said,  “We  have  faster,  more  advanced  machines  in  NJROTC  [than  the  rest 
of  the  school],  and  they’re  maintained  better.” 

15James  Page  of  Apple  Computer,  Inc.,  wrote,  "The  lab  is  maintained  in  a  clean,  or¬ 
derly  fashion,  and  the  students  are  taught  to  value  this  resource  ....  By  visiting  the  lab 
at  Lemoore  and  contrasting  it  with  other  labs,  one  can  see  this  aspect  of  how  the  lab  is 
managed.  My  own  personal  reaction,  when  visiting  the  lab,  was  to  treat  the  equip¬ 
ment  carefully,  and  with  respect ....  When  the  time  spent  in  the  lab  is  made  part  of 
the  overall  discipline  and  respect  that  is  part  of  the  JROTC  program,  then  I  believe  that 
there  can  be  no  question  but  that  it  will  be  far  more  successful  than  other  labs  where 
the  equipment  is  not  treated  with  as  much  respect  and  in  such  a  professional  man¬ 
ner.” 
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Attract  More  and  Better  Students 

The  Naval  Science  Instructors  at  Lemoore  and  Sanger  stated  that,  in 
their  opinion,  the  NMIS  is  a  principal  factor  that  enables  the  NJROTC 
program  to  attract  students. 

The  system  brings  students  to  the  program— and  we  need  num¬ 
bers— and  it  brings  better-quality  students. 


In  most  programs,  there’s  a  pyramid  by  year,  with  fewer  seniors  at 
the  top  because  they  need  to  satisfy  academic  requirements  for 
graduation.  Traditional  NJROTC  programs  have  a  tough  time  get¬ 
ting  seniors  to  take  the  course  as  an  elective.  Last  year,  we  had  55 
seniors,  and  this  year  we’ll  have  the  same  or  even  more. 

The  NJROTC  program  at  [our  school]  draws  top  students  as  well  as 
non-college  bound  students. 

Anecdotal  evidence  appears  to  support  these  claims.  As  reported 
above  (Chapter  Four),  65  seniors  had  applied  by  early  August  for  the 
30  slots  in  the  NJROTC  section  of  the  American  Government  course 
at  Sanger,  and  more  applications  were  expected  to  be  received  before 
the  school  year  began.  A  student  reported  that  students  want  to 
enroll  in  the  NJROTC  section  of  the  course  because  it  covers  more 
material,  it  is  more  enjoyable  than  the  sections  taught  in  the 
traditional  manner,  and  students  in  it  gain  computer  skills  that  they 
believe  will  give  them  an  advantage  in  their  endeavors  after  they 
graduate. 

We  met  NJROTC  cadets  at  Sanger  who  included  student  body  offi¬ 
cers,  the  just-elected  student  body  president,  and  a  member  of  the 
school’s  Academic  Decathlon  team.  We  were  advised  that  a  majority 
of  one  school’s  valedictorians  and  salutatorians  in  recent  years  have 
been  NJROTC  cadets. 

“Extracurricular”  Uses  of  the  NMIS 

Students  use  the  NMIS  outside  of  NJROTC  class  time  for  both  aca¬ 
demic  and  extra-curricular  purposes.  About  one-fourth  of  the  stu¬ 
dents  in  the  focus  groups  reported  that  they  use  the  NMIS  during 
“first  period”  (or  “home  room”  or  “study  hall”),  during  lunch,  or  after 
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school.  Upperclassmen  more  than  freshman  said  they  used  the  sys¬ 
tem  outside  of  normal  class  hours. 

They  said  they  use  the  system  to  catch  up  when  they  miss  an 
NJROTC  class,  to  review  material  that  was  covered  in  class  which 
they  did  not  understand,  and  to  complete  assignments.  Over  half  of 
the  students  reported  they  have  used  the  system  for  academic  work 
in  non-NJROTC  courses.  For  example,  several  students  said  they  had 
used  the  Magazine  Data  Base  (see  the  subsection  “Links  to  External 
Information  Sources,”  Chapter  Two)  for  non-NJROTC  courses  such 
as  biology  and  English. 

In  addition,  NMIS  equipment  is  used  regularly  for  NJROTC  activities 
such  as  the  following: 

•  The  cadet  leader  of  the  Drill  Team  communicates  with  team 
members  by  putting  announcements,  drill  instructions,  etc.,  on 
the  system;  team  members  retrieve  the  information  during  their 
classes,  before  or  after  school,  etc. 

•  Cadets  use  the  auxiliary  computer  to  plan  the  Military  Awards 
Dinner. 

•  The  cadet  commanders  of  the  Color  Guard  and  Drill  Team 
record  dates  of  performances,  participation  by  team  members, 
ribbons  earned,  and  date  awarded. 

•  The  Rifle  Team  uses  a  spreadsheet  to  keep  scores  and  record  the 
points  that  are  used  to  determine  who  gets  a  varsity  letter. 

•  The  auxiliary  computer  is  used  for  Navy  JROTC  finances,  to 
complete  Navy  reports  (using  templates,  so  that  the  information 
is  printed  in  the  correct  position  on  Navy  forms),  NJROTC  inven¬ 
tory  (equipment,  uniforms,  etc.),  and  administrative  correspon¬ 
dence. 

•  A  template  in  the  word  processing  program  that  is  loaded  on  the 
auxiliary  computer  is  used  to  enter  names,  dates,  and  other  data 
on  preprinted  award  certificates;  an  instructor  commented,  “It’s 
faster  than  typing  them,  and  they  look  a  lot  better  off  the  laser 
printer  [than  a  typewriter]  because  we  can  use  better  fonts.” 


Chapter  Six 

DISCUSSION 


In  this  chapter,  we  posit  alternative  models  to  explain  students’ 
achievement  on  the  tests  that  were  administered  by  RAND.  We  ob¬ 
serve  that  the  NMIS  appears  to  encourage  and  structure  users’  be¬ 
haviors  in  ways  that  are  likely  to  lead  to  positive  outcomes,  and  this 
aspect  of  the  system  presents  opportunities  for  pedagogical  innova¬ 
tion.  Finally,  we  note  that  students’  and  instructors’  comments  sug¬ 
gest  that  implementation  issues  are  likely  to  be  critical  to  the  success 
of  the  system  at  other  NJROTC  locations. 

STUDENTS’  ACHIEVEMENT  ON  THE  RAND  TESTS 

In  all  three  of  the  courses  that  participated  in  this  study,  the  average 
scores  of  the  students  in  the  NMIS  groups  increased  more  from  pre¬ 
test  to  post-test  than  did  scores  of  students  in  the  Traditional  groups. 
In  the  NS  1  course  at  Lemoore,  the  NMIS  group's  score  increased 
32.8  percent,  whereas  the  Traditional  group’s  score  increased  13.5 
percent.  At  Sanger,  the  NMIS  group’s  score  was  up  23.1  percent,  and 
the  Traditional  group’s  score  increased  only  15.9  percent.  In 
Sanger’s  American  Government  course,  the  NMIS  group’s  score 
improved  29.7  percent,  but  the  Traditional  group’s  score  declined  7.3 
percent. 

These  findings  support  the  assertion  that  the  NMIS  makes  a  positive 
difference  in  students’  achievement.  Moreover,  the  findings  in  the 
NS  1  course  at  Sanger  suggest  that  the  NMIS  is  equally  effective  for 
groups  of  students  who  read  with  different  levels  of  proficiency.  On 
average,  Hispanic  students  scored  significantly  lower  than  did 
students  of  other  ethnicities  on  the  CAT/5  reading  scale;  in  addition, 
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Hispanic  students  in  the  NMIS  group  had  lower  scores,  on  average, 
than  did  Hispanic  students  in  the  Traditional  group.  Even  so,  the 
scores  of  the  Hispanic  students  in  the  NMIS  group  increased  23.4 
percent  from  pre-test  to  post-test,  a  slightly  higher  gain  than  was 
achieved  by  other  students  in  the  NMIS  group.  In  contrast,  the 
scores  of  Hispanic  students  in  the  Traditional  group  increased  only 
15.5  percent,  comparable  to  other  students  in  the  Traditional  group. 

The  improvement  in  the  students’  scores  during  this  study  from  the 
pre-test  to  the  post-test  suggests  that  the  NMIS  “raises  the  floor”  to 
some  minimum  level  of  achievement.  It  can  be  posited  that  this  out¬ 
come  can  be  explained  by  learning  theory  (Kaput,  1989).  That  is, 
multiple  representations  of  material  promote  better  retention  on  the 
part  of  students,  and,  as  described  above,  students  in  NMIS  class¬ 
rooms  hear  the  lesson  delivered  orally,  see  the  material  presented  vi¬ 
sually,  and,  concurrently,  take  notes  on  the  material  manually  (by 
typing  on  a  keyboard). 

However,  the  findings  may  be  explained  other  ways.  We  did  not 
identify  pre-test  and  post-test  scores  by  student  (to  protect  students’ 
privacy),  and,  although  the  rate  of  attrition  was  comparable  in  both 
groups  at  both  schools,  a  disproportionate  number  of  students  with 
low  pre-test  scores  may  have  dropped  from  the  NMIS  groups  before 
the  post-test.  Eliminating  them  from  the  post-test  would  have  raised 
the  groups’  scores  regardless  of  the  efficacy  of  the  system  (assuming 
that  these  students’  post-test  scores  would  have  been  low  relative  to 
the  other  students’  scores). 

It  also  is  possible  that  there  may  have  been  an  “instructor  effect”  at 
Sanger  where  one  instructor  taught  the  NMIS  sections  of  the  NS  1 
and  American  Government  courses  while  other  instructors  taught 
the  Traditional  sections.  However,  the  difference  in  the  achievement 
of  the  NS  1  groups  at  Lemoore  cannot  be  attributed  to  an  instructor 
effect  because  the  same  person  taught  all  four  sections  of  the  course. 
It  is  possible  this  individual’s  level  of  energy  varied  during  the  day 
(the  Traditional  group  was  taught  during  the  last  class  period),  but, 
during  our  visits  to  the  classes,  we  did  not  observe  any  indication 
that  this  might  be  the  case. 

On  the  other  hand,  it  is  possible  that  the  effect  of  the  NMIS  on  stu¬ 
dents’  scores  is  understated  in  this  study.  If  the  period  of  study  had 
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been  longer,  the  difference  between  the  groups’  scores  might  have 
favored  the  NMIS  group  even  more  strongly.  There  are  two  principal 
reasons: 

First,  the  system  was  new  to  the  students  and,  indeed,  to  one  of  the 
instructors  using  it  as  well.  In  general,  students  stated  during  the  fo¬ 
cus  groups  that  the  system  was  easy  to  learn,  but  they  also  said  it  got 
easier  to  use  as  they  gained  experience  with  it.  Students  may  have 
been  building  system-related  skills  during  much  of  the  approxi¬ 
mately  ten-week  period  that  elapsed  between  the  pre-test  and  the 
post-test.  They  may  have  been  using  the  system  effectively  only  near 
the  end  of  this  study. 

Second,  there  may  have  been  a  “ceiling”  on  the  range  of  the  NS  1  test 
scores  because  the  first  ten  weeks  of  the  course  may  have  covered 
material  that  was  already  known  to  the  students  or  was  learned  rela¬ 
tively  easily  by  them.1  It  is  likely  that  the  range  of  variation  in  the 
scores  would  have  been  greater  had  this  study  extended  over  a  longer 
period,  so  that  the  test  could  have  covered  more  extensive  and, 
perhaps,  more  challenging  course  material. 

The  findings  relative  to  the  American  Government  course  are  indeed 
surprising.  The  improvement  of  the  NMIS  group’s  scores  from  pre¬ 
test  to  post-test  is  consistent  with  the  improvement  of  the  NMIS 
groups  in  the  NS  1  course.  However,  it  seems  highly  unusual  that  the 
scores  of  the  students  in  the  Traditional  group  would  decline  from 
pre-test  to  post-test.  One  plausible  explanation  is  that  students  who 
made  high  scores  on  the  pre-test  transferred  out  of  the  Traditional 
sections.  A  second  possible  explanation  is  instructor  effect.  A  third 
possible  explanation  was  offered  by  the  instructor  of  the  Traditional 
group  who  said  that  students  do  not  do  well  on  tests  that  are  not  part 
of  their  course  grade.  “They  probably  weren’t  motivated  because 


The  relatively  high  scores  on  the  NS  1  pre-test  suggest  that  some  of  the  material  was 
familiar  to  students.  During  the  focus  groups,  students  commented  that  the  pre-test 
and  post-test  were  not  easy.  However,  many  of  the  students  replied  that  they  had 
family  members  in  naval  or  military  service.  It  is  possible  that  these  students  learned 
some  of  the  course  material  through  association  with  family  members.  During  a  focus 
group  session,  one  student  commented  in  reference  to  an  NS  1  class  that  was  taught 
with  traditional  methods  of  instruction  during  a  previous  school  year,  “Over  half  of  the 
book  was  boring.  It  was  easy  to  read,  but  you  didn’t  want  to  read  it.  It  was  clear  and 
well  organized.  But  you  could  answer  the  questions  without  reading  it.” 


48  Evaluation  of  the  NJROTC  Multimedia  Instructional  System 


they  weren’t  accountable.”  However,  this  explanation  is  not  wholly 
satisfying  because  the  scores  of  only  one  group  declined,  but  all  of 
the  students  participating  in  this  study  were  told  at  the  start  of  each 
test  that  their  score  would  not  affect  their  course  grade.  A  fourth 
possible  explanation  might  be  inferred  from  the  fact  that  the  stu¬ 
dents  in  the  NJROTC  section  of  the  American  Government  course 
were  self-selected.  As  noted  above  (Chapter  Four),  students  com¬ 
peted  to  be  in  the  NJROTC  section.  This  fact  can  be  interpreted  two 
ways.  On  one  hand,  a  non-NJROTC  instructor  volunteered,  "Kids 
take  that  [NJROTC]  class  because  it’s  easier.  There’s  no  homework. 
They  study  less  material.  I  make  them  think.  There,  it’s  rote  learn¬ 
ing.  That’s  not  what  they  need  for  college.”  On  the  other  hand, 
anecdotal  evidence  suggests  that  achievement-oriented  students  en¬ 
roll  in  NJROTC  (see  the  subsection  “Attract  More  and  Better 
Students,”  Chapter  Five).  In  addition,  a  student  commented  that 
students  choose  the  NJROTC  section  of  the  American  Government 
course  because  it  is  more  interesting,  covers  more  material,  and 
equips  them  with  skills  they  expect  to  use  in  the  future. 


THE  NMIS  SHAPES  USERS’  BEHAVIORS 

It  also  can  be  argued  that  the  system  predisposes  its  users  to  behav¬ 
iors  (such  as  taking  notes  on  course  materials  and  lectures)  that  fa¬ 
cilitate  learning.  It  does  this  by  providing  rewards,  by  imposing  con¬ 
straints,  and  by  freeing  users  from  limitations  that  are  inherent  in 
traditional  classroom  settings. 

For  example,  instructors  use  the  testing  module  of  the  NMIS  be¬ 
cause,  among  other  reasons,  it  scores  tests  and  eliminates  many 
hours  of  tedious  work.  They  adhere  to  the  lecture  outline,  appar¬ 
ently,  because  it  is  readily  available,  it  is  helpful,  and  students  expect 
them  to  follow  it.  Instructors  said  they  like  to  use  the  NMIS  because 
it  frees  them  from  the  blackboard  and  media  devices  at  the  front  of 
the  classroom,  and,  thereby,  enables  them  to  walk  around  the  room 
during  their  lecture,  monitor  the  class,  and  assist  individual  students 
more  fully  than  in  traditional  classroom  settings. 

Students  tend  to  look  up  definitions  in  NMIS  course  materials  be¬ 
cause  the  hypertext  links  are  easy  and  “fun”  to  use.  They  tend  to  re¬ 
fer  back  to  points  they  missed  during  a  lecture  because  they  can  eas¬ 
ily  “scroll”  the  Instructor’s  Notes.  In  contrast,  they  reported  that  in 
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traditional  class  settings  they  generally  do  not  make  the  effort  to  look 
up  unfamiliar  words  or  follow  up  on  a  missed  point.  During  most  of 
the  class  period,  students  remain  seated  at  their  desks,  at  work  on 
assigned  tasks,  in  part,  we  suggest,  because  the  system  conducts  an 
active  “dialog”  that  is  engaging  and  motivating,  and  in  part  because 
the  students  expect  to  be  held  accountable — in  classes  using  the 
NMIS,  it  is  readily  apparent  who  is  not  taking  notes,  and  students 
know  their  instructors  are  able  to  review  their  work  more  easily  than 
teachers  in  non-NMIS  classes.  Similarly,  the  “Jeopardy  Game”  en¬ 
gages  students  during  the  discussion  period  because  it  is  “fun"  and 
because  they  are  aware  that  each  of  them  has  an  equal  chance  of  be¬ 
ing  called  upon  to  answer  a  question.  Students  reported  that  they 
prefer  to  take  tests  using  the  NMIS  because  they  value  the  immediate 
feedback  it  provides,  and,  indeed,  that  feedback  appears  to  be  a 
source  of  positive  reinforcement.2 

The  ability  of  the  system  to  shape  its  users’  behaviors  has  positive 
implications.  Presumably,  “canned”  lectures  can  be  developed  that 
would  ensure  that  lessons  are  delivered  with  a  minimum  level  of 
proficiency,  almost  regardless  of  an  instructor’s  professional  skill 
and/or  level  of  motivation.3  Similarly,  it  may  be  possible  to  develop 
new  ways  for  students  to  interact  with  the  system;  ways  that  would 
engage  the  students  more  fully  in  the  curriculum. 

OPPORTUNITY  FOR  PEDAGOGICAL  INNOVATION 

In  discussions  with  instructors  who  were  using  the  system  and  in  our 
observations  of  classes,  we  noted  that  classes  using  the  NMIS  follow 
a  traditional  didactic  instructional  approach.  That  is,  although  the 
NMIS  functions  to  some  extent  as  a  tool  for  more  effective  class 
management,  it  has  had  relatively  little  effect  on  pedagogy,  and 
instruction  remains  primarily  teacher- centered. 


2During  our  observations  of  the  classes,  we  noted  that  students  appeared  to  be  ac¬ 
tively  engaged  by  the  system.  For  example,  the  immediate  feedback  provided  by  the 
testing  module  elicited  exclamations  (which,  usually,  the  students  made  softly  to 
themselves)  such  as,  “‘B’ — of  course!”  and  "I  knew  it  was  ‘D’!”  Many  students  also  re¬ 
sponded  to  the  feedback  with  “body  English.” 

3 

It  is  assumed  that  instructors  could  enhance  the  "canned”  lectures  by  tailoring  them 
to  their  individual  preferences  and  incorporating  supplementary  materials. 
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Students  do  not  use  textbooks,  but  they  do  read  material  on  their 
computers  that  has  been  scanned  into  the  system  from  traditional 
textbooks.4  Instructors  rely  mainly  on  a  lecture  format  to  deliver 
information.  They  use  the  NMIS  to  organize  lecture  information  for 
the  students  and  help  them  stay  on  track  (through  the  mechanisms 
of  lecture  outlines  and  note-taking  templates). 

Cuban  (1988)  notes  that  successful  school  reforms  generally  incorpo¬ 
rate  changes  that  “try  to  make  what  already  exists  more  efficient  and 
more  effective.”  The  NMIS  may  be  effective  because  instructors  use 
it  to  improve  the  delivery  of  curriculum  in  a  familiar,  traditional 
fashion.  Taking  a  longer  view,  we  question  whether  it  is  also  possible 
to  take  advantage  of  the  potential  in  the  system  to  make  learning 
more  student-centered  and  teach  students  how  to  be  more  respon¬ 
sible  for  their  learning.  Reference,  for  example,  the  initiatives  of  the 
Institute  for  Learning  Technologies,  Teachers  College,  Columbia 
University,  which  are  designed  to  enable  students  to  access  teaching 
resources  independent  of  teachers  (Columbia,  1995a).  Also  refer¬ 
ence  "The  Edison  Project,”  which  is  being  used  in  the  Chemistry  De¬ 
partment  at  Columbia  with  the  basic  premise  that  “students  should 
be  actively  involved  in  their  own  education.  'We  want  to  challenge 
and  change  the  time-honored  format  of  the  professor  who  stands  at 
the  blackboard,  with  knowledge  passing  in  one  direction’” 
(Columbia,  1995b). 

We  question,  for  example,  whether  the  study  skills  students  use  with 
the  system  (e.g.,  taking  notes  in  outline  form,  referring  to  notes  while 
writing  essays)  transfer  to  other  subjects,  or  whether  they  relieve  stu¬ 
dents  of  learning  to  apply  the  proper  organizational  structures  to 
their  work.  Are  there  coaching  strategies  (cf.  Brown  et  al.,  1988)  that 
might  be  introduced  as  students  grow  familiar  with  the  system  to  en¬ 
sure  that  they  assimilate  the  study  skills  and  make  them  a  natural 
part  of  their  study  habits? 

Another  concern  we  have  is  that  the  system  has  yet  to  fully  access  the 
information  resources  that  are  available  to  students  electronically 


4However,  the  NMIS  course  materials  are  not  merely  scanned  versions  of  the  text¬ 
books.  For  example,  the  system  includes  hypertext  links  that  connect  words  in  the  text 
to  the  glossary.  This  mechanism,  as  noted  above,  increases  the  likelihood  that  stu¬ 
dents  will  look  up  unfamiliar  words. 
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(e.g.,  books,  articles,  films,  documentaries,  and  personal  interviews 
via  the  Internet  and  on  compact  disks),  and  help  students  acquire 
the  personal  command  of  information  that  is  becoming  a  mandate  in 
modern  society.  Linking  the  NMIS  with  the  Internet  opens  myriad 
possibilities  for  students  to  access  and  apply  information.  To  take 
advantage  of  this  opportunity  requires  what  Cuban  (1988)  labels 
“second  order  changes”  which  “introduce  new  goals,  structures,  and 
roles  that  transform  familiar  ways  of  doing  things  into  new  ways  of 
solving  persistent  problems.”  Instructors  would  need  to  rethink  their 
roles  in  the  classroom,  the  organization  of  their  curricula,  and  the 
structure  of  their  teaching. 

Second  order  changes  are  difficult  to  accomplish  and  require  new 
problem  definitions  and  solutions,  but  we  believe  they  offer  sub¬ 
stantial  benefits  in  equipping  students  with  skills  that  they  will  need 
in  the  future. 

THE  IMPORTANCE  OF  SYSTEM  IMPLEMENTATION  AND 
SUPPORT 

This  study  did  not  investigate  in  depth  the  motivation,  training,  and 
on-going  support  provided  to  users.  However,  students’  and  instruc¬ 
tors’  comments  indicate  that  implementation  issues  are  likely  to  be 
critical  to  the  success  of  the  system  at  other  NJROTC  locations. 

Information  systems  literature  explains  that  users’  decisions  whether 
to  adopt  a  system  are  influenced  by  their  initial  perceptions  of  it 
(Lucas,  1981),  and  these  perceptions  are  strongly  influenced  by 
users’  individual  characteristics  (Devin,  1994).  In  other  words,  a  sys¬ 
tems’  objective  costs  and  benefits  may  be  less  important  than  the 
users’  subjective  perceptions  of  them.  Typical  of  users  in  general, 
NMIS  users  appear  to  have  had  polar  impressions  of  the  system. 

You  see  a  computer,  you  say  it’s  going  to  be  fun,  not  a  boring  lec¬ 
ture. 

Initially,  it  might  look  as  though  it  burdens  one.  Once  you  under¬ 
stand  it,  then  it  eases  the  teacher’s  workload. 
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The  first  hurdle  is  to  get  prospective  users  to  try  a  system.  Studies 
show  that  users’  perceptions  are  influenced  by  advertising,  their  per¬ 
ceptions  of  peers’  experiences  with  the  system  or  a  comparable  sys¬ 
tem,  etc.  (cf.  Devin,  1994). 

The  second  hurdle  is  to  implement  the  system  in  a  manner  that  in¬ 
creases  the  likelihood  that  users’  initial  experiences  with  it  will  be 
positive.  This  phenomenon  is  explained  in  the  literature  (cf.  Devin, 
1994);  the  practical  consequences  are  described  eloquently  by  an  in¬ 
structor  and  a  student  who  participated  in  this  study: 

A  board,  markers,  and  erasers  never  fail  you.  When  the  system  hic¬ 
cups,  you  need  proper  support  or  you’ll  say,  “That’s  all  I  needed.  I’ll 
never  let  it  do  that  to  me  again!” 

I  mess  around  on  the  computer  at  home,  and  it  gets  old  getting 
trapped  [by  a  system  fault] ,  and  not  being  able  to  get  out.  When  you 
do,  you  don’t  want  to  touch  it  again.  You  distrust  it. 

As  the  first  quote  indicates,  one  way  to  ameliorate  the  frustrations 
that  often  are  encountered  by  new  users  is  to  provide  adequate  on¬ 
going  support.  It  will  be  recalled  that  instructors  themselves  com¬ 
mented  on  the  importance  of  system  support  (see  the  subsection 
“Would  Instructors  Recommend  the  NMIS  to  Other  NJROTC  Units?” 
Chapter  Five). 

We  suggest  that  the  vehicle  used  for  system  support  might  also  be 
used  beneficially  to  promote  and  disseminate  innovations.  In  gen¬ 
eral,  users  employ  a  system  at  first  as  a  substitute  for  a  procedure 
with  which  they  are  familiar,  then,  as  they  gain  experience  with  it, 
they  discover  new  ways  to  use  it  (cf.  Pape  and  Thoresen,  1990;  Sand- 
strom,  1987;  Szlichcinski,  1983).  Consistent  with  this  phenomenon, 
we  observed  that  students  and  instructors  appeared  to  integrate  the 
NMIS  more  fully  into  their  work  as  they  gained  familiarity  with  it.  In 
the  main,  students’  innovations  appeared  to  be  related  to  extracur¬ 
ricular  activities.  Instructors  were  taking  steps  toward  new  ap¬ 
proaches  to  teaching.  For  example,  they  found  with  the  NMIS  they 
could  walk  around  the  classroom  and  interact  more  effectively  with 
students  than  when  they  used  traditional  methods  of  instruction. 
They  found  that  the  sound  of  typing— both  continuous  typing  and 
the  absence  of  typing — indicates  whether  students’  understand 
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and/or  are  attentive  to  the  lecture,  and  they  adjust  the  pacing  of  the 
lesson  accordingly.  We  applaud  these  steps,  and  we  suggest,  as 
discussed  above  (see  the  subsection  "Opportunity  for  Pedagogical 
Innovation,’’  Chapter  Six),  that  important  innovations  are  likely  to  be 
found  by  exploring  new  approaches  that  enable  and  encourage 
students  to  use  the  NMIS  to  a  larger  extent  on  their  own  initiative. 


Chapter  Seven 

CONCLUSIONS 


The  data  collected  during  this  study  indicate  that  the  NMIS  has  posi¬ 
tive  effects  on  students’  achievement,  on  their  attitudes,  and  on  in¬ 
structors’  workload.  Our  interpretation  of  the  data  suggests  that 
there  exists  a  tight  interweave  among  the  students’  academic 
achievement,  their  attitudes  toward  their  courses,  and  the  instruc¬ 
tors’  workload. 

It  appears  that  the  NMIS  is  efficacious  because  it  facilitates  the  use  of 
methods  of  instruction  that  educators  have  found  to  be  effective — 
methods  (such  as  using  multiple  media,  providing  immediate  feed¬ 
back,  and  helping  students  individually)  that  often  are  difficult  to 
apply  in  classrooms  today  because  of  large  class  sizes,  administrative 
burdens  placed  on  teachers,  reliance  on  teaching  tools  that  are  in¬ 
convenient  to  use,  and  challenging  academic  skills,  expectations, 
and  behaviors  of  many  of  the  students.  The  system  also  appears  to 
be  effective  because  it  engages  students  interactively,  and,  in  the 
process,  it  transforms  tasks  that  students  find  onerous  in  traditional 
classes  into  game-like  activities. 

IMPLICATIONS  FOR  CNET 

In  this  concluding  chapter,  we  discuss  the  potential  for  extending  the 
use  of  the  NMIS  beyond  its  current  applications.  The  cost  of  hard¬ 
ware,  software,  and  training  is,  of  course,  a  primary  consideration  in 
the  decision  whether  to  expand  the  use  of  information  technology  in 
the  NJROTC  curricula;  but,  in  addition,  we  suggest  that  the  evalua¬ 
tion  should  also  weigh  the  value  of  the  following  implications  of  the 
system. 
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A  Virtual  Presence  in  NJROTC  Units 

NMIS  technology  enables  CNET  to  maintain  a  “virtual  presence”  in 
NJROTC  classrooms  and,  thereby,  “raise  the  floor”  for  academic 
achievement  to  some  minimum  level,  particularly  at  locations  where 
students  require  remedial  assistance.  The  four  principal  enablers  are 
the  Instructors’  Notes  component  of  the  system,  course  supplements 
containing  current  material,  tests,  and  “portfolio”  sampling. 

Instructors’  Notes  encourage  instructors  to  deliver  the  curriculum  in 
a  consistent  and  professional  manner  that  incorporates  multi-media. 
We  posit  that  many  NJROTC  instructors  are  likely  to  follow  the  Notes 
more  closely  than  hardcopy  course  materials  because  the  Notes  re¬ 
quire  minimal  preparation  on  the  part  of  the  instructors.  The  Notes 
may  be  particularly  helpful  to  individuals  who  have  not  taught  high 
school  students  previously.  They  also  may  serve  to  motivate  more 
experienced  instructors  who  are  experiencing  "burn  out.”  Yet,  they 
can  be  tailored  by  innovative  instructors. 

From  many  students’  perspective,  out-of-date  course  material  often 
implies  that  a  course  is  not  relevant  and,  consequently,  not  worth 
their  attention.  Although  it  is  difficult  and  expensive  in  a  rapidly 
changing  world  to  maintain  up-to-date  hardcopy  texts  and  materials, 
it  is  relatively  easy  and  inexpensive  to  revise  and  supplement  NMIS 
courses  electronically  (via  the  Internet,  email,  or  diskettes).  Instruc¬ 
tors  could  incorporate  electronically  transmitted  updates  into  their 
courses  with  relatively  little  effort. 

CNET  could  distribute  tests  electronically  to  NJROTC  units.  The  tests 
could  cover  the  core  concepts  that  CNET  determines  instructors 
should  teach  in  a  course,  and  they  could  be  issued  immediately  be¬ 
fore  being  given  in  order  to  minimize  “teaching  to  the  test.”  The 
tests  would  be  scored  by  NMIS  software,  and  the  results,  by  student, 
could  be  forwarded  electronically  to  CNET.  A  program  at  CNET 
could  consolidate  the  data,  analyze  it,  and  compare  students’ 
achievement  by  NJROTC  unit.  This  technique  would  enable  CNET  to 


'a  Naval  Science  Instructor  suggested  that  items  of  interest  to  NJROTC  students  could 
be  distributed  via  the  Internet  in  the  same  manner  that  the  Navy  (OPNAV)  distributes 
the  "Yellow  Sheet"  via  electronic  mail  to  inform  personnel  of  key  articles  appearing  in 
major  newspapers. 
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evaluate  the  effectiveness  of  the  instruction  at  each  unit  using  objec¬ 
tive  statistical  techniques  similar  to  those  used  in  conjunction  with 
national  tests  such  as  the  SAT,  GMAT,  LSAT,  etc. 

The  NMIS  enables  CNET  to  evaluate  the  curriculum  and  the  NJROTC 
units’  performance  in  new  ways.  For  example,  it  might  borrow  from 
the  “portfolio  assessment”  method  that  has  been  adopted  by  various 
school  districts,  and  randomly  sample  and  review  the  files  that  con¬ 
tain  students’  reading  and  lecture  notes,  essays,  and  homework.  This 
“assessment”  might  be  done,  at  least  in  part,  by  a  computer  program 
that  would  automatically  draw  files  at  random  from  NJROTC  sites 
and  compile  statistics  about  them. 

An  Opportunity  to  Promote  Pedagogical  Innovation 

As  we  have  suggested  in  this  report,  the  NMIS  creates  an  environ¬ 
ment  that  is  conducive  to  innovation.  CNET  might  use  the  system  as 
a  platform  for  pedagogical  innovation  that  could  “raise  the  ceiling” 
for  academic  achievement  and  effectively  challenge  learners  with 
differing  levels  of  ability.  It  appears  that  innovative  use  of  informa¬ 
tion  systems  technology  would  enable  NJROTC  units  to  serve  as  a 
role  model  for  other  departments  of  their  schools,  and  enhance 
NJROTC  interaction  with  the  school  community.  Anecdotal  evi¬ 
dence  suggests  these  initiatives  would  also  serve  to  help  attract  more 
and  better  students  to  the  NJROTC  program. 

Potential  Cost  Savings 

We  did  not  investigate  issues  relative  to  the  logistics  of  publishing 
and  distributing  course  materials.  However,  the  NMIS  technology 
would  enable  CNET  to  publish  and  distribute  “courseware”  electron¬ 
ically  (and  update  the  materials,  as  well,  as  discussed  above).  It  is 
probable  that  the  costs  associated  with  electronic  courseware  would 
be  markedly  less  than  the  costs  associated  with  traditional  textbooks, 
etc.  Indeed,  electronic  publishing  and  distribution  may  offset,  to  a 
significant  degree,  the  cost  of  the  system. 
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SUGGESTIONS  FOR  FUTURE  RESEARCH 

As  stated  above,  this  was  an  exploratory  study.  The  findings  indicate 
that  a  more  rigorous  study  of  the  NMIS  would  be  worthwhile.  Such  a 
study  would  identify  the  “critical  success  factors”  for  the  NMIS  in 
terms  of  the: 

1.  Training  and  technical  assistance  needs  of  users  (both  students 
and  instructors), 

2.  Implementation  requirements,  particularly  in  educational  settings 
where  the  students,  instructors,  and  infrastructure  differ  from 
those  at  Lemoore  and  Sanger,  and 

3.  Human/computer  interface  factors  that  help  users  learn  and  use 
the  system. 

Learning  more  about  the  training  and  technical  support  needs  of 
NMIS  users  (both  students  and  instructors)  would  be  a  valuable  aid 
to  designing  effective  implementation  procedures. 

A  related  research  agenda  concerns  hardware  and  software,  i.e., 
whether  other  configurations  of  the  system  might  improve  its 
price/performance  ratio. 


Appendix  A 

THE  NMIS  CLASSROOM 


The  NMIS  classrooms  at  Lemoore  and  at  Sanger  differ  somewhat; 
Figure  A.1  shows  a  general  schematic  of  the  installations. 

A  television  monitor  is  located  at  the  front  of  the  room;  the  large 
screen  (“home  theater”  size)  enables  all  of  the  students  in  the  class¬ 
room  to  see  the  material  displayed  on  it. 

The  instructor’s  workstation,  also  at  the  front  of  the  room,  includes  a 
computer  (i.e.,  computer,  monitor,  keyboard,  and  mouse),  videocas¬ 
sette  player,  laser  disc  player,  and  a  switch  that  connects  the  output 


Figure  A.  1 — Diagram  of  a  Typical  NMIS  Classroom 
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from  either  the  video  or  the  laser  disc  player  to  the  television  moni¬ 
tor.  The  instructor’s  machine  has  supervisory  control  over  the  stu¬ 
dents’  machines,  that  is,  instructors  can  download /upload  material 
to/from  the  students’  machines,  and  they  can  lock  the  students’  key¬ 
boards. 

The  students’  machines  sit  four  to  a  table  on  seven  tables,  which  are 
approximately  eight  feet  long  by  two  feet  wide.  At  both  schools,  the 
care  with  which  the  equipment  and  the  classroom  are  maintained  is 
readily  apparent  to  students  and  visitors  alike  (cf.  the  subsection 
“Impact  of  the  NMIS  on  School  Culture,”  Chapter  Five).  The  com¬ 
puters  are  aligned  in  precise  rows.  When  they  are  not  being  used,  the 
keyboards  are  tucked  neatly  under  the  overhang  of  the  monitor,  and 
the  mice  (most  of  them),  are  placed  just  to  the  right  of  the  keyboard. 
Students  are  expected  to  return  their  equipment  to  this  position  at 
the  end  of  the  class.  Instructors  check  the  equipment  daily  for  dam¬ 
age.  Comments  made  by  at  both  schools  by  NJROTC  instructors, 
school  administrators,  and  students  suggest  that  it  is  because  of  this 
discipline  that  the  NJROTC  NMIS  classrooms  have  fewer  mainte¬ 
nance  problems  and  are  more  enjoyable  to  use  than  the  computer 
labs  in  other  departments  of  the  schools. 

Extensive  remodeling  was  not  required  for  the  installation.  Cables 
connecting  the  students’  machines  to  the  power  source  and  the  in¬ 
structors’  computer  are  tied  underneath  the  tables,  largely  out  of 
sight,  and  continue  along  the  walls,  relatively  inconspicuously. 


_ Appendix  B 

RESULTS  OF  THE  NAVAL  SCIENCE  1  TEST  AT 

LEMOORE 


The  pre-test  was  taken  by  81  students.  The  post-test  was  completed 
by  79  students. 

Students  in  the  NMIS  group  scored,  on  average,  31.7  points  on  the 
pre-test,  and  students  in  the  Traditional  group  scored  37.9  points. 
The  difference  in  the  groups’  pre-test  scores  is  highly  significant  (p  = 
0.0002).  That  is,  the  probability  is  zero  that  the  scores  of  two  groups 
are  from  the  same  population. 

The  average  score  on  the  post- test  was  42.1  for  the  NMIS  group  and 
43.0  for  the  Traditional  group.  The  difference  in  the  post-test  scores 
is  not  statistically  significant  (p  =  0.6390).  In  other  words,  the  groups 
were  relatively  comparable  in  terms  of  their  post-test  scores. 

During  the  term,  the  average  score  of  the  NMIS  group  increased  32.8 
percent  (i.e.,  up  10.4  points,  from  31.7  to  42.1)  whereas  the  average 
score  of  the  Traditional  group  increased  13.5  percent  (i.e.,  up  5.1 
points,  from  37.9  to  43.0).  The  6.2  point  “advantage”  of  the 
Traditional  group  on  the  pre-test  (37.9  versus  31.7)  was  narrowed  to 
0.9  point  on  the  post-test  (43.0  versus  42.1). 

There  is  no  significant  difference  in  the  Lemoore  students’  scores  on 
the  NS  1  test  when  the  scores  are  partitioned  by  school  year  within 
group,  or  by  gender  within  group.  The  scores  of  the  Hispanic 
students  at  Sanger,  on  average,  were  lower  than  those  of  other  ethnic 
groups  on  the  pre-test  and  the  post-test.  It  is  not  meaningful  to 
partition  the  Lemoore  data  by  ethnicity  within  group  because  there 


61 


62  Evaluation  of  the  NJROTC  Multimedia  Instructional  System 


were  not  sufficient  numbers  of  students  in  each  category  to  provide 
statistical  reliability. 

Figure  B.l  shows  the  distribution  of  pre-test  and  post-test  scores. 
Four  graphs  are  shown.  The  two  topmost  graphs  illustrate  the 
distribution  of  the  pre-test  scores  for  the  two  groups:  The  NMIS 
group  is  on  the  left  side  of  the  figure  and  the  Traditional  group  on  the 
right  side.  Scores  are  shown  with  a  histogram,  where  the  bars  cover 
the  five-point  intervals  that  are  displayed  on  the  horizontal  axis.  The 
vertical  axis  indicates  the  percentage  of  the  group  in  a  given  interval. 
For  example,  approximately  36  percent  of  the  students  in  the  NMIS 
group  had  pre-test  scores  between  30  and  35  points.  A  normal  curve 
has  been  fitted  to  the  data  and  is  superimposed  on  the  histogram.  A 
vertical  line  marks  the  mean  of  the  distribution. 
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Figure  B.l — NS  1  Pre-Test  and  Post-Test  Scores  at  Lemoore  High  School 
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The  two  lower  graphs  show  the  distribution  of  the  post-test  scores  for 
the  two  groups.  The  NMIS  group  is  on  the  left  and  the  Traditional 
group  is  on  the  right.  For  ease  of  comparison,  the  bottom  graphs 
show  a  gray  vertical  line  and  a  gray  curve  which  represent  the 
group’s  pre-test  mean  and  normal  distribution. 

The  graphs  illustrate  that  (1)  the  Traditional  group  had  higher  scores 
on  the  pre-test,  (2)  scores  for  both  groups  improved  from  the  pre-  to 
the  post-test,  and  (3)  the  scores  of  the  NMIS  group  improved  more 
than  did  the  scores  of  the  Traditional  group,  i.e.,  the  curve  of  the 
NMIS  group  shifted  a  greater  distance  to  the  right. 
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RESULTS  OF  THE  NAVAL  SCIENCE  1  TEST  AT  SANGER 


The  pre-test  was  taken  by  104  students,  and  83  students  took  the 
post-test.  The  same  NS  1  tests  were  used  at  both  Sanger  and 
Lemoore,  but  scores  for  Sanger  should  not  be  compared  directly  with 
scores  for  Lemoore  because  somewhat  different  material  was  cov¬ 
ered  at  the  two  schools. 

The  average  test  scores  on  the  pre-test  were  33.7  for  the  NMIS  group 
and  33.4  for  the  Traditional  group.  The  difference  is  not  statistically 
significant  (p  =  0.8672). 

The  average  scores  on  the  post-test  were  41.5  for  the  NMIS  group 
and  38.7  for  the  Traditional  group.  The  difference  is  statistically  sig¬ 
nificant  below  the  10  percent  level  (p  =  0.0679).  In  other  words,  there 
is  a  less  than  10  percent  probability  that  the  2.8  point  margin 
favoring  the  NMIS  group  is  a  chance  event. 

The  average  score  in  the  NMIS  group  increased  23.1  percent  (7.8 
points)  from  the  pre-test  to  the  post-test.  The  Traditional  group's  av¬ 
erage  score  increased  15.9  percent  (5.3  points). 

There  are  no  statistically  significant  differences  in  the  NS  1  test 
scores  by  grade  level  or  by  gender  within  the  groups.  Table  C.l,  be¬ 
low,  reports  that  the  NS  1  test  scores  of  the  Hispanic  students  in  the 
NMIS  group  increased  23.4  percent  from  the  pre-test  to  the  post-test, 
slightly  more  than  the  22. 1  percent  increase  in  the  scores  of  the  other 
students  in  the  NMIS  group.  In  contrast,  the  scores  of  the  Hispanic 
students  in  the  Traditional  group  improved  15.5  percent,  a  rate  com¬ 
parable  to  that  of  the  other  students  in  the  Traditional  group. 
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Table  C.l 

Sanger  NS  1  Scores  by  Ethnicity 


Group 

Subgroup 

Pre-Test 

Post-Test 

Change 

NMIS 

Hispanic 

32.07 

39.59 

+7.52 

23.4% 

All  other 

36.21 

44.20 

+7.99 

22.1% 

total 

33.71 

41.46 

+7.75 

23.0% 

Traditional3 

Hispanic 

30.96 

35.75 

+4.79 

15.5% 

All  other 

36.20 

41.82 

+5.62 

15.5% 

total 

33.45 

38.65 

+5.20 

15.5% 

aData  are  incomplete  regarding  the  ethnicity  of  some  students  in  the 
Traditional  group  who  took  the  pre-test.  These  data  have  been  esti¬ 
mated  in  part. 


Figure  C.l  shows  the  distribution  of  pre-test  and  post-test  scores. 
Four  graphs  are  shown.  The  two  topmost  graphs  illustrate  the  distri¬ 
bution  of  the  pre-test  scores  for  the  two  groups:  The  NMIS  group  is 
on  the  left  side  of  the  figure  and  the  Traditional  group  on  the  right 
side.  Scores  are  shown  with  a  histogram,  where  the  bars  cover  the 
five-point  intervals  that  are  displayed  on  the  horizontal  axis.  A  nor¬ 
mal  curve  has  been  fitted  to  the  data  and  is  superimposed  on  the  his¬ 
togram.  The  vertical  axis  indicates  the  percentage  of  the  group  in  a 
given  interval.  For  example,  approximately  20  percent  of  the  stu¬ 
dents  in  the  Traditional  group  had  pre-test  scores  between  40  and  45 
points. 

The  two  lower  graphs  show  the  distribution  of  the  post-test  scores  for 
the  two  groups.  The  NMIS  group  is  on  the  left  and  the  Traditional 
group  is  on  the  right.  For  ease  of  comparison,  the  bottom  graphs 
show  a  gray  vertical  line  and  a  gray  curve  which  represent  the 
group’s  pre-test  mean  and  normal  distribution. 

The  graphs  illustrate  that  (1)  the  groups’  pre-test  scores  were  compa¬ 
rable,  (2)  scores  for  both  groups  improved  from  the  pre-  to  the  post¬ 
test,  and  (3)  the  post-test  scores  of  the  NMIS  group  improved  more 
than  did  the  scores  of  the  Traditional  group,  that  is,  the  curve  for  the 
NMIS  group  shifted  further  to  the  right  than  did  the  curve  for  the 
Traditional  group. 
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Figure  C.l — NS  1  Pre-Test  and  Post-Test  Scores  at  Sanger  High  School 


_ Appendix  D 

RESULTS  OF  THE  AMERICAN  GOVERNMENT  TEST 


The  pre-test  included  94  students  and  the  post-test  included  82  stu¬ 
dents. 

The  average  score  of  the  NMIS  group  on  the  pre-test  was  17.2.  The 
Traditional  group’s  average  score  was  19.2.  The  difference  is  statis¬ 
tically  significant  below  the  10  percent  level  (p  =  0.0796). 

On  the  post-test,  the  NMIS  group’s  average  score  increased  29.7  per¬ 
cent  (5.1  points).  The  Traditional  group’s  average  score  decreased 
7.3  percent  (-1.4  points).  The  difference  is  statistically  significant 
(p  =  0.0015). 

There  are  no  statistically  significant  differences  in  the  test  scores 
when  they  are  partitioned  by  gender  or  ethnicity. 

Figure  D.l  shows  the  distribution  of  pre-test  and  post-test  scores. 
Four  graphs  are  shown.  The  two  topmost  graphs  illustrate  the  distri¬ 
bution  of  the  pre-test  scores  for  the  two  groups:  The  NMIS  group  is 
on  the  left  side  of  the  figure  and  the  Traditional  group  on  the  right 
side.  Scores  are  shown  with  a  histogram,  where  the  bars  cover  the 
five-point  intervals  that  are  displayed  on  the  horizontal  axis.  A  nor¬ 
mal  curve  has  been  fitted  to  the  data  and  is  superimposed  on  the  his¬ 
togram.  The  vertical  axis  indicates  the  percentage  of  the  group  in  a 
given  interval.  For  example,  approximately  43  percent  of  the  stu¬ 
dents  in  the  NMIS  group  had  pre-test  scores  between  15  and  20 
points. 
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Figure  D.l — American  Government  Pre-Test  and  Post-Test  Scores 
at  Sanger  High  School 

The  two  lower  graphs  show  the  distribution  of  the  post-test  scores  for 
the  two  groups.  The  NMIS  group  is  on  the  left  and  the  Traditional 
group  is  on  the  right.  For  ease  of  comparison,  the  bottom  graphs 
show  a  gray  vertical  line  and  a  gray  curve  which  represent  the 
group’s  pre-test  mean  and  normal  distribution. 

The  graphs  illustrate  that  (1)  pre-test  scores  were  higher  for  the 
Traditional  group  than  the  NMIS  group,  (2)  post-test  scores  of  the 
NMIS  group  are  higher  than  its  pre-test  scores,  and,  in  fact,  are 
skewed  toward  the  higher  end  of  the  scale,  but  (3)  post-test  scores  of 
the  Traditional  group  declined  from  the  pre-test  level  while 
maintaining,  approximately,  the  same  distribution,  which  suggests 
there  is  no  anomaly  such  as  the  top  students  leaving  the  class. 


_ Appendix  E 

TASKS  STUDENTS  MUST  LEARN  TO  USE  THE  NMIS 


•  Tasks  to  read  course  material 
—  Bring  up  menu  bar 

—  Use  scroll  bar 

—  Click  highlighted  text  to  bring  up  pictures,  notes,  definitions 
—  Go  to  next  page 

—  Go  to  a  specific  page  in  a  document 
—  Return  to  main  folder 

•  Additional  tasks  to  write  notes  and  answer  the  questions  at  the 
end  of  each  unit 

—  Copy  from  a  HyperCard  file 

—  Paste  into  a  Claris  Works  file 

—  Use  the  tab  to  make  an  outline 

—  “Find”  text  in  a  document 

•  Additional  tasks  to  write  essays  using  the  notes  file(s) 

—  Use  a  split  screen 

—  Be  able  to  highlight  a  window 

•  Additional  tasks  associated  with  tests 
—  Enter  the  test  program 
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—  Click  answer 
—  “Mark”  an  answer 
—  Go  to  next  question 

—  Return  to  prior  questions  that  were  not  answered 
—  End  test 

•  Optional,  advanced  skills1 

—  Create  and  use  a  HyperCard  stack 

—  Use  the  HyperCard  tool  box 

—  Make  HyperCard  buttons  and  fields 

—  Use  the  scanner  and,  perhaps,  telecommunications 
software 


llhese  skills  are  acquired  by  students  who  elect  to  use  the  NMIS  to  give  presentations, 
e.g.,  to  report  a  team  project  to  the  class.  Typically,  they  are  obtained  via  coaching  by 
the  instructor  and/or  trial-and-error  practice. 
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